[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:49 dist-test-slave-xltq krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:49 dist-test-slave-xltq krb5kdc[399](info): set up 2 sockets
Sep 27 00:38:49 dist-test-slave-xltq krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:38:50.896672 392 mini_kdc.cc:121] Time spent starting KDC: real 1.859s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:38:50 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:38:51.023327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:51.135025 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:51.135361 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:51.135421 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:51.139665 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:51.139743 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:51.139771 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:51.139793 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:51.139813 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:51.145866 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:51.147033 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:51.148167 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:51.158233 412 server_base.cc:1034] running on GCE node
W20260927 00:38:51.158247 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.158241 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.158330 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:51.159247 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:51.160701 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:51.161895 412 hybrid_clock.cc:648] HybridClock initialized: now 1790469531161802 us; error 114 us; skew 500 ppm
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:51.164901 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:38:51.165928 412 webserver.cc:460] Webserver started at http://127.0.98.62:37567/ using document root <none> and password file <none>
I20260927 00:38:51.166514 412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:51.166587 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:51.166810 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:51.169064 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "ca5d7a1a52ca4851912cded04798bd72"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b7eb05dfbd8b0a3c49e1f419cac7f700"
server_key_iv: "5aa3d18558a794e08dd6e7eeea8e4650"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.169634 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca5d7a1a52ca4851912cded04798bd72"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b7eb05dfbd8b0a3c49e1f419cac7f700"
server_key_iv: "5aa3d18558a794e08dd6e7eeea8e4650"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.174007 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:51.176810 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.178172 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:51.178303 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ca5d7a1a52ca4851912cded04798bd72"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b7eb05dfbd8b0a3c49e1f419cac7f700"
server_key_iv: "5aa3d18558a794e08dd6e7eeea8e4650"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.178417 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:51.191116 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:51.194739 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:51.194955 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:51.202209 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46827
I20260927 00:38:51.202225 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46827 every 8 connection(s)
I20260927 00:38:51.203231 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:51.206379 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:51.210253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 412
I20260927 00:38:51.210407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:38:51.211740 392 external_mini_cluster.cc:1428] Setting key 9dc12ff597a1201663cbde33e0eddd2a
I20260927 00:38:51.212441 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: Bootstrap starting.
I20260927 00:38:51.215226 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:51.216267 480 log.cc:824] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:51.218539 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: No bootstrap required, opened a new log
I20260927 00:38:51.223109 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 00:38:51.223352 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:51.223412 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5d7a1a52ca4851912cded04798bd72, State: Initialized, Role: FOLLOWER
I20260927 00:38:51.223847 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 00:38:51.223976 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:51.224056 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:51.224160 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:51.225399 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 00:38:51.225766 480 leader_election.cc:302] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca5d7a1a52ca4851912cded04798bd72; no voters:
I20260927 00:38:51.226123 480 leader_election.cc:288] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:51.226245 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:51.226399 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [term 1 LEADER]: Becoming Leader. State: Replica: ca5d7a1a52ca4851912cded04798bd72, State: Running, Role: LEADER
I20260927 00:38:51.226716 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260927 00:38:51.227519 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:51.228732 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca5d7a1a52ca4851912cded04798bd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
I20260927 00:38:51.228870 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:51.229270 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca5d7a1a52ca4851912cded04798bd72. Latest consensus state: current_term: 1 leader_uuid: "ca5d7a1a52ca4851912cded04798bd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca5d7a1a52ca4851912cded04798bd72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
I20260927 00:38:51.229367 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72 [sys.catalog]: This master's current role is: LEADER
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 00:38:51.233075 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:51.233165 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:51.233320 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:51.234375 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:51.240089 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.214859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.0.98.62:46827)
0927 00:38:51.215576 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:51.215597 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:51.216840 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 00:38:51.220902 (+ 4062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:51.220930 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:51.221399 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:51.221840 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:51.224253 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.225714 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.227355 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.227955 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.228126 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.232674 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:51.232682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:51.232718 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:51.232774 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:51.235158 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.235989 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.235996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.236002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.236157 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.236731 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.236735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.236736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.237221 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:51.237412 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:51.238415 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0927 00:38:51.238723 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":201,"threads_started":1}
I20260927 00:38:51.242067 494 catalog_manager.cc:1347] Generated new cluster ID: 8dc5b71ad2a84fd8b613939366dbaf34
I20260927 00:38:51.242147 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:51.263190 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:51.264354 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:51.273705 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca5d7a1a52ca4851912cded04798bd72: Generated new TSK 0
I20260927 00:38:51.274509 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:38:51.311645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:51.426111 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:51.426422 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:51.426467 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:51.430963 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:51.431051 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:51.431115 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:38:51.436651 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:51.437785 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:51.438838 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:51.449491 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.449465 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:51.449555 506 server_base.cc:1034] running on GCE node
W20260927 00:38:51.449476 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:51.450289 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:51.450989 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:51.452260 506 hybrid_clock.cc:648] HybridClock initialized: now 1790469531452232 us; error 37 us; skew 500 ppm
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:51.455366 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:38:51.456394 506 webserver.cc:460] Webserver started at http://127.0.98.1:44581/ using document root <none> and password file <none>
I20260927 00:38:51.456979 506 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:51.457063 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:51.457283 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:51.459450 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "c25a2d0aea2f474f8890763c6766a333"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "347bda82c433d47e256456f4a146c6ca"
server_key_iv: "4b70b7ae8f7fdf0aaa09af228b36af1b"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.460037 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c25a2d0aea2f474f8890763c6766a333"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "347bda82c433d47e256456f4a146c6ca"
server_key_iv: "4b70b7ae8f7fdf0aaa09af228b36af1b"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.464051 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:38:51.466835 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.468604 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:51.468750 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "c25a2d0aea2f474f8890763c6766a333"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "347bda82c433d47e256456f4a146c6ca"
server_key_iv: "4b70b7ae8f7fdf0aaa09af228b36af1b"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.468879 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:51.486758 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:51.490314 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:51.490538 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:51.491163 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:51.492095 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:51.492151 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.492234 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:51.492264 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.502058 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35865
I20260927 00:38:51.502082 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35865 every 8 connection(s)
I20260927 00:38:51.503050 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:38:51.508117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 506
I20260927 00:38:51.508268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:38:51.508527 392 external_mini_cluster.cc:1428] Setting key 1e51f0a8ee19fe540f4e7cde8b6cece0
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:38:51.518467 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.505202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32991 (local address 127.0.98.62:46827)
0927 00:38:51.505328 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:51.505331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:51.506603 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 00:38:51.508001 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:51.508004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:51.508054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:51.508129 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:51.509625 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.510274 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.511471 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.511714 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.511808 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.514508 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:51.514512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:51.514514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:51.514543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:51.515971 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.516652 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.516655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.516657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.516747 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.517146 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.517148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.517150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.517407 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:51.517502 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:51.518088 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 00:38:51.518246 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260927 00:38:51.519306 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.505579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.1:32991)
0927 00:38:51.506049 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:51.506097 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:51.507755 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:51.508382 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:51.508393 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:51.508918 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:51.509473 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.509492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.510399 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.510402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.511349 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.511359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.511868 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.511872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.512036 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.512621 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:51.512638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:51.514278 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:51.516109 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.516117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.516138 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.516533 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.516846 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.516849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.516851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.517056 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.517513 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:51.517521 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:51.517834 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 00:38:51.518013 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:38:51.518387 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:38:51.520668 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 00:38:51.520941 635 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:51.521471 635 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 00:38:51.523317 444 ts_manager.cc:194] Registered new tserver with Master: c25a2d0aea2f474f8890763c6766a333 (127.0.98.1:35865)
I20260927 00:38:51.524645 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:38:51.536087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:51.653513 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:51.653833 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:51.653896 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:51.658166 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:51.658248 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:51.658358 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:38:51.663985 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:51.665357 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:51.666488 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:51.676680 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.676725 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.676725 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:51.677187 640 server_base.cc:1034] running on GCE node
I20260927 00:38:51.677614 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:51.678351 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:51.679526 640 hybrid_clock.cc:648] HybridClock initialized: now 1790469531679502 us; error 38 us; skew 500 ppm
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:51.682381 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:38:51.683400 640 webserver.cc:460] Webserver started at http://127.0.98.2:33613/ using document root <none> and password file <none>
I20260927 00:38:51.683988 640 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:51.684056 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:51.684319 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:51.686489 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "cdbbc86a5d7349eab573d12f94d8cc0e"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b839d7a960bd6e5b4939e32e5dc9f35e"
server_key_iv: "3608db822b722c7182b3bfea76337330"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.687069 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cdbbc86a5d7349eab573d12f94d8cc0e"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b839d7a960bd6e5b4939e32e5dc9f35e"
server_key_iv: "3608db822b722c7182b3bfea76337330"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.691186 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:38:51.694386 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.695928 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:51.696090 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "cdbbc86a5d7349eab573d12f94d8cc0e"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "b839d7a960bd6e5b4939e32e5dc9f35e"
server_key_iv: "3608db822b722c7182b3bfea76337330"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.696269 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:51.711297 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:51.714769 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:51.714962 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:51.715593 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:51.716557 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:51.716610 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.716668 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:51.716692 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.726680 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36557
I20260927 00:38:51.726719 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36557 every 8 connection(s)
I20260927 00:38:51.727705 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:38:51.733168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 640
I20260927 00:38:51.733291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:38:51.733525 392 external_mini_cluster.cc:1428] Setting key 9213fd834a9744716313c90477e3d974
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:38:51.743893 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.729791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43291 (local address 127.0.98.62:46827)
0927 00:38:51.729937 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:51.729941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:51.730936 (+ 995us) server_negotiation.cc:407] Connection header received
0927 00:38:51.732416 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:51.732419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:51.732472 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:51.732558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:51.733937 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.734488 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.735876 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.736277 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.736429 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.739165 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:51.739168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:51.739170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:51.739207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:51.741232 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.741878 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.741882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.741883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.741989 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.742435 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.742439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.742441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.742788 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:51.742905 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:51.743624 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 00:38:51.743761 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:38:51.744858 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.730236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.2:43291)
0927 00:38:51.730762 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:51.730814 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:51.732117 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:51.732757 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:51.732765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:51.733158 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:51.733765 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.733787 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.734626 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.734630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.735729 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.735740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.736481 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.736485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.736722 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.737396 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:51.737414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:51.738952 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:51.741401 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.741408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.741424 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.741738 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.742132 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.742135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.742137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.742325 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.742924 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:51.742930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:51.743333 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 00:38:51.743547 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:38:51.743886 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:38:51.746336 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 00:38:51.746703 769 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:51.747282 769 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 00:38:51.748612 444 ts_manager.cc:194] Registered new tserver with Master: cdbbc86a5d7349eab573d12f94d8cc0e (127.0.98.2:36557)
I20260927 00:38:51.749400 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:38:51.765288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:51.885632 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:51.885936 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:51.885980 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:51.890265 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:51.890350 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:51.890416 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:38:51.896368 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:51.897482 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:51.898483 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:51.909016 774 server_base.cc:1034] running on GCE node
W20260927 00:38:51.909008 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.909261 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:51.908983 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:51.909829 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:51.910566 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:51.911755 774 hybrid_clock.cc:648] HybridClock initialized: now 1790469531911741 us; error 31 us; skew 500 ppm
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:51.915189 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:38:51.916291 774 webserver.cc:460] Webserver started at http://127.0.98.3:39203/ using document root <none> and password file <none>
I20260927 00:38:51.916920 774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:51.916997 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:51.917204 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:51.919453 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "1da9208a93fe43d18a3723038f9ab21a"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "082801b2800c390514082f0c5346f0c9"
server_key_iv: "3475642c216dc110bd75ac12baf78960"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.920076 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1da9208a93fe43d18a3723038f9ab21a"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "082801b2800c390514082f0c5346f0c9"
server_key_iv: "3475642c216dc110bd75ac12baf78960"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.924338 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:38:51.927345 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.928946 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:51.929088 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "1da9208a93fe43d18a3723038f9ab21a"
format_stamp: "Formatted at 2026-09-27 00:38:51 on dist-test-slave-xltq"
server_key: "082801b2800c390514082f0c5346f0c9"
server_key_iv: "3475642c216dc110bd75ac12baf78960"
server_key_version: "encryptionkey@0"
I20260927 00:38:51.929211 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:51.948012 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:51.951640 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:51.951850 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:51.952526 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:51.953382 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:51.953428 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.953480 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:51.953506 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:51.964483 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38107
I20260927 00:38:51.964592 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38107 every 8 connection(s)
I20260927 00:38:51.965488 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:38:51.972441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 774
I20260927 00:38:51.972560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:38:51.972826 392 external_mini_cluster.cc:1428] Setting key 22022b98aa26132f3e220526796cdae3
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:51.980744 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.967472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48763 (local address 127.0.98.62:46827)
0927 00:38:51.967695 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:51.967699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:51.968629 (+ 930us) server_negotiation.cc:407] Connection header received
0927 00:38:51.970081 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:51.970084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:51.970138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:51.970298 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:51.971813 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.972492 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.973925 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.974230 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.974325 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.977201 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:51.977205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:51.977207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:51.977236 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:51.978565 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.979158 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.979162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.979164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.979244 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:51.979570 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:51.979573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:51.979575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:51.979777 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:51.979864 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:51.980441 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 00:38:51.980571 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 00:38:51.981601 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:51.967932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.3:48763)
0927 00:38:51.968459 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:51.968516 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:51.969789 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:51.970530 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:51.970541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:51.970967 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:51.971625 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.971642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.972637 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.972641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.973812 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:51.973823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:51.974404 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:51.974408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:51.974604 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:51.975239 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:51.975256 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:51.977005 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:51.978752 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.978759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.978774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.979054 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.979329 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:51.979332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:51.979334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:51.979482 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:51.979877 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:51.979884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:51.980168 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 00:38:51.980397 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 00:38:51.980749 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
I20260927 00:38:51.983016 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260927 00:38:51.983366 903 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:51.983911 903 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260927 00:38:51.985209 444 ts_manager.cc:194] Registered new tserver with Master: 1da9208a93fe43d18a3723038f9ab21a (127.0.98.3:38107)
I20260927 00:38:51.985788 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48763
I20260927 00:38:51.987993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:38:51 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:52.015350 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:52.004946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.0.98.1:35865)
0927 00:38:52.005303 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:52.005319 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:52.005346 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:38:52.005414 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:52.005420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:52.005625 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:52.005756 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:52.006514 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:52.007354 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:52.008363 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:52.008778 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:52.008874 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:52.011558 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:52.011570 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:52.011586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:52.011615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:52.013316 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:52.013840 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:52.013847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:52.013848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:52.013951 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:52.014290 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:52.014293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:52.014294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:52.014565 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:52.014765 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:52.014936 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:38:52.015078 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":168,"threads_started":1}
W20260927 00:38:52.016470 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38506
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:52.043315 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:52.034387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.0.98.1:35865)
0927 00:38:52.034572 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:52.034587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:52.034740 (+ 153us) server_negotiation.cc:407] Connection header received
0927 00:38:52.034890 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:52.034893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:52.034935 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:52.034999 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:52.035751 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:52.036389 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:52.037264 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:52.037500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:52.037591 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:52.040309 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:52.040312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:52.040314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:52.040345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:52.041625 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:52.042066 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:52.042070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:52.042071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:52.042152 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:52.042415 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:52.042419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:52.042421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:52.042602 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:52.042719 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:52.043015 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 00:38:52.043139 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:38:52.044648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 506
I20260927 00:38:52.050005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 640
I20260927 00:38:52.057037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 774
I20260927 00:38:52.061774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 412
2026-09-27T00:38:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3055 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[917](info): set up 2 sockets
Sep 27 00:38:52 dist-test-slave-xltq krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 00:38:53.019815 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:38:53.143918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:53.254688 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:53.255000 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:53.255059 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:53.259207 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:53.259289 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:53.259317 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:53.259337 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:53.259359 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:53.264955 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:53.266093 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:53.267140 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:53.275885 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.275871 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.275950 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:53.276131 930 server_base.cc:1034] running on GCE node
I20260927 00:38:53.276763 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:53.277992 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:53.279167 930 hybrid_clock.cc:648] HybridClock initialized: now 1790469533279137 us; error 43 us; skew 500 ppm
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:53.288376 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:38:53.289409 930 webserver.cc:460] Webserver started at http://127.0.98.62:38293/ using document root <none> and password file <none>
I20260927 00:38:53.290011 930 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:53.290086 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:53.290294 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:53.292492 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "b50288604d54443e9a20d8a85f459be8"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d7b4ff40e40925c5b65a1b93e1905072"
server_key_iv: "e3ea771c4da53bf73b6480c35356099e"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.293068 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "b50288604d54443e9a20d8a85f459be8"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d7b4ff40e40925c5b65a1b93e1905072"
server_key_iv: "e3ea771c4da53bf73b6480c35356099e"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.297052 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:38:53.299608 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.301090 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:38:53.301218 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "b50288604d54443e9a20d8a85f459be8"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d7b4ff40e40925c5b65a1b93e1905072"
server_key_iv: "e3ea771c4da53bf73b6480c35356099e"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.301333 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:53.313540 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:53.316915 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:53.317083 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:53.323815 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36239
I20260927 00:38:53.323832 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36239 every 8 connection(s)
I20260927 00:38:53.324887 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:53.327893 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:53.330619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 930
I20260927 00:38:53.330720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:38:53.330972 392 external_mini_cluster.cc:1428] Setting key fd9ed56ace230fef9c7031b9cbba7a58
I20260927 00:38:53.334920 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: Bootstrap starting.
I20260927 00:38:53.337710 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:53.338577 998 log.cc:824] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:53.341171 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: No bootstrap required, opened a new log
I20260927 00:38:53.344457 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } }
I20260927 00:38:53.344738 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:53.344798 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50288604d54443e9a20d8a85f459be8, State: Initialized, Role: FOLLOWER
I20260927 00:38:53.345291 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } }
I20260927 00:38:53.345445 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:53.345530 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:53.345650 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:53.346706 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } }
I20260927 00:38:53.347071 998 leader_election.cc:302] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b50288604d54443e9a20d8a85f459be8; no voters:
I20260927 00:38:53.347165 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:53.332426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.0.98.62:36239)
0927 00:38:53.333321 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:53.333403 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:53.333445 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:38:53.334128 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:53.334148 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:53.334546 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:53.334869 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:53.335788 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.336863 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.338007 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.338593 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.338756 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:53.341716 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:53.341722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:53.341777 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:53.341807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:53.343928 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.344492 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.344497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.344502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.344621 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.344930 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.344934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.344936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.345365 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:53.345577 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:53.345865 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 00:38:53.346337 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":580,"threads_started":1}
I20260927 00:38:53.347451 998 leader_election.cc:288] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:53.347544 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:53.347766 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [term 1 LEADER]: Becoming Leader. State: Replica: b50288604d54443e9a20d8a85f459be8, State: Running, Role: LEADER
I20260927 00:38:53.348143 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } }
I20260927 00:38:53.348740 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:53.350047 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b50288604d54443e9a20d8a85f459be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } } }
I20260927 00:38:53.350168 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:53.350545 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b50288604d54443e9a20d8a85f459be8. Latest consensus state: current_term: 1 leader_uuid: "b50288604d54443e9a20d8a85f459be8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50288604d54443e9a20d8a85f459be8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36239 } } }
I20260927 00:38:53.350634 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8 [sys.catalog]: This master's current role is: LEADER
W20260927 00:38:53.353742 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:38:53.353808 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:38:53.353965 1014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:53.354905 1014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:53.360822 1014 catalog_manager.cc:1347] Generated new cluster ID: f858f8404d7840f49fe654e1761ba367
I20260927 00:38:53.360915 1014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:53.404567 1014 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:53.406014 1014 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:53.412827 1014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b50288604d54443e9a20d8a85f459be8: Generated new TSK 0
I20260927 00:38:53.413764 1014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:38:53.453389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:53.565420 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:53.565857 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:53.565959 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:53.570575 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:53.570662 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:53.570729 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:38:53.576426 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:53.577548 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:53.578619 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:38:53.588871 1024 server_base.cc:1034] running on GCE node
W20260927 00:38:53.588811 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.588827 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.588827 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:53.589545 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:53.590248 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:53.591414 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790469533591387 us; error 45 us; skew 500 ppm
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:53.594247 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:38:53.595191 1024 webserver.cc:460] Webserver started at http://127.0.98.1:32793/ using document root <none> and password file <none>
I20260927 00:38:53.595762 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:53.595829 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:53.596014 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:53.598147 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ad6793c82ea40ceafc9e0d13b6da77c"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "1ee0ced1fd21fb39a80d818d2f6cad54"
server_key_iv: "d2603015984a512cd2caf8639cee4fde"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.598706 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ad6793c82ea40ceafc9e0d13b6da77c"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "1ee0ced1fd21fb39a80d818d2f6cad54"
server_key_iv: "d2603015984a512cd2caf8639cee4fde"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.602540 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:53.605237 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.606683 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:38:53.606846 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "9ad6793c82ea40ceafc9e0d13b6da77c"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "1ee0ced1fd21fb39a80d818d2f6cad54"
server_key_iv: "d2603015984a512cd2caf8639cee4fde"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.606967 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:53.623526 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:53.627224 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:53.627449 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:53.628121 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:53.629076 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:53.629130 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.629190 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:53.629227 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.638837 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46473
I20260927 00:38:53.638813 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46473 every 8 connection(s)
I20260927 00:38:53.639896 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:38:53.649690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1024
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:53.649780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:38:53.650032 392 external_mini_cluster.cc:1428] Setting key 34cae4fbd70bd1138227aba70546877e
I20260927 00:38:53.654449 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:53.641802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43289 (local address 127.0.98.62:36239)
0927 00:38:53.641955 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:53.641960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:53.642835 (+ 875us) server_negotiation.cc:407] Connection header received
0927 00:38:53.644179 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:53.644182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:53.644270 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:53.644344 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:53.645575 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.646108 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.647482 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.647717 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.647808 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:53.650477 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:53.650480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:53.650482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:53.650509 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:53.652089 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.652704 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.652707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.652709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.652799 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.653273 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.653275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.653276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.653463 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:53.653544 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:53.654197 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 00:38:53.654310 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:38:53.655627 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:53.642217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36239 (local address 127.0.98.1:43289)
0927 00:38:53.642679 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:53.642725 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:53.643955 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:53.644524 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:53.644532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:53.644919 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:53.645426 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:53.645441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.646230 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.646233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:53.647358 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:53.647369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.647864 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.647868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:53.648044 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:53.648654 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:53.648671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:53.650279 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:53.652297 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:53.652302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:53.652317 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:53.652600 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:53.652914 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:53.652917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:53.652919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:53.653172 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:53.653561 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:53.653568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:53.653923 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 00:38:53.654161 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 00:38:53.654521 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260927 00:38:53.656944 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36239
I20260927 00:38:53.657191 1153 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:53.657722 1153 heartbeater.cc:511] Master 127.0.98.62:36239 requested a full tablet report, sending...
I20260927 00:38:53.659214 959 ts_manager.cc:194] Registered new tserver with Master: 9ad6793c82ea40ceafc9e0d13b6da77c (127.0.98.1:46473)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:38:53.660389 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:38:53.679589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:53.793321 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:53.793612 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:53.793674 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:53.797894 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:53.797978 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:53.798063 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:38:53.803817 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:53.804986 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:53.806011 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:53.815819 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.815821 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:53.815848 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:53.816273 1158 server_base.cc:1034] running on GCE node
I20260927 00:38:53.816723 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:53.817408 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:53.818573 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790469533818533 us; error 61 us; skew 500 ppm
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:53.821400 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:38:53.822382 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37995/ using document root <none> and password file <none>
I20260927 00:38:53.822913 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:53.822984 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:53.823155 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:53.825429 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "62fd52b2df584fe78249250db9ca1746"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d1d9142b20a87c7f87323c7a62a5d7ce"
server_key_iv: "cbc1dc8f8acab2c80757da96910398dd"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.825984 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62fd52b2df584fe78249250db9ca1746"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d1d9142b20a87c7f87323c7a62a5d7ce"
server_key_iv: "cbc1dc8f8acab2c80757da96910398dd"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.829931 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:53.832646 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.833992 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:53.834105 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "62fd52b2df584fe78249250db9ca1746"
format_stamp: "Formatted at 2026-09-27 00:38:53 on dist-test-slave-xltq"
server_key: "d1d9142b20a87c7f87323c7a62a5d7ce"
server_key_iv: "cbc1dc8f8acab2c80757da96910398dd"
server_key_version: "encryptionkey@0"
I20260927 00:38:53.834206 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:53.850572 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:53.853928 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:53.854096 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:53.854668 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:53.855506 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:53.855554 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.855595 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:53.855614 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:53.864369 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32951
I20260927 00:38:53.864399 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32951 every 8 connection(s)
I20260927 00:38:53.865301 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:38:53.865661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1158
I20260927 00:38:53.865779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:38:53.866062 392 external_mini_cluster.cc:1428] Setting key fbf33e010a825655ad181650488ffde4
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 00:38:53 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:53.883263 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:53.867297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45799 (local address 127.0.98.62:36239)
0927 00:38:53.867438 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:53.867442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:53.868612 (+ 1170us) server_negotiation.cc:407] Connection header received
0927 00:38:53.870140 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:53.870144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:53.870202 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:53.870293 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:53.871930 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.872508 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.874238 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.874455 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.874537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:53.879162 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:53.879167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:53.879169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:53.879203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:53.880928 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.881477 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.881481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.881483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.881585 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:53.881943 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:53.881947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:53.881949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:53.882238 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:53.882361 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:53.882925 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 00:38:53.883069 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:38:53.884035 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:53.867873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36239 (local address 127.0.98.2:45799)
0927 00:38:53.868429 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:53.868501 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:53.869915 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:53.870548 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:53.870559 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:53.871038 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:53.871759 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:53.871783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.872653 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.872657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:53.874102 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:53.874115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:53.874618 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:53.874622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:53.874836 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:53.875445 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:53.875468 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:53.878912 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:53.881083 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:53.881089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:53.881104 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:53.881374 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:53.881676 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:53.881678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:53.881679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:53.881848 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:53.882349 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:53.882354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:53.882629 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 00:38:53.882798 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:38:53.883125 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
I20260927 00:38:53.885278 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36239
I20260927 00:38:53.885540 1287 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:53.886063 1287 heartbeater.cc:511] Master 127.0.98.62:36239 requested a full tablet report, sending...
I20260927 00:38:53.887122 959 ts_manager.cc:194] Registered new tserver with Master: 62fd52b2df584fe78249250db9ca1746 (127.0.98.2:32951)
I20260927 00:38:53.887769 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:38:53.894886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:54.008515 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:54.008816 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:54.008862 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:54.013247 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:54.013316 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:54.013387 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:38:54.018901 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36239
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:54.019982 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:54.021059 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:54.031333 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:54.031464 1292 server_base.cc:1034] running on GCE node
W20260927 00:38:54.031334 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:54.031335 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:54.032027 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:54.032785 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:54.033955 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790469534033924 us; error 48 us; skew 500 ppm
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:54.037055 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:38:54.038074 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35687/ using document root <none> and password file <none>
I20260927 00:38:54.038652 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:54.038724 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:54.038959 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:54.041293 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0a874e97b01410ab228958653e95ae4"
format_stamp: "Formatted at 2026-09-27 00:38:54 on dist-test-slave-xltq"
server_key: "94890cdefe28f60712189ba3ec6bb658"
server_key_iv: "e5ef8a6b388c7b4fd83df552bc84462c"
server_key_version: "encryptionkey@0"
I20260927 00:38:54.041888 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0a874e97b01410ab228958653e95ae4"
format_stamp: "Formatted at 2026-09-27 00:38:54 on dist-test-slave-xltq"
server_key: "94890cdefe28f60712189ba3ec6bb658"
server_key_iv: "e5ef8a6b388c7b4fd83df552bc84462c"
server_key_version: "encryptionkey@0"
I20260927 00:38:54.046149 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:54.048950 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:54.050459 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:54.050585 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "e0a874e97b01410ab228958653e95ae4"
format_stamp: "Formatted at 2026-09-27 00:38:54 on dist-test-slave-xltq"
server_key: "94890cdefe28f60712189ba3ec6bb658"
server_key_iv: "e5ef8a6b388c7b4fd83df552bc84462c"
server_key_version: "encryptionkey@0"
I20260927 00:38:54.050697 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:54.066962 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:54.070153 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:54.070331 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:54.070917 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:38:54.071759 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:54.071807 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:54.071858 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:54.071885 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:54.080492 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44263
I20260927 00:38:54.080528 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44263 every 8 connection(s)
I20260927 00:38:54.081444 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:54.091207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1292
I20260927 00:38:54.091327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:38:54.091593 392 external_mini_cluster.cc:1428] Setting key bea326f4d402dc2d3832b189c6419c72
I20260927 00:38:54.095361 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.083399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41069 (local address 127.0.98.62:36239)
0927 00:38:54.083587 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.083592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.084482 (+ 890us) server_negotiation.cc:407] Connection header received
0927 00:38:54.085806 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.085809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.085864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.085940 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:54.087141 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.087695 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.088923 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.089167 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.089252 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.091814 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:54.091818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:54.091821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:54.091851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:54.093426 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.094018 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.094021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.094023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.094097 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.094378 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.094380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.094382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.094572 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:54.094647 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.095137 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.095239 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:38:54.096186 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.083864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36239 (local address 127.0.98.3:41069)
0927 00:38:54.084337 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.084380 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.085608 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.086092 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.086099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.086453 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:54.086986 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.087003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.087827 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.087829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.088809 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.088818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.089290 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.089293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.089433 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.090046 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:54.090062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:54.091596 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:54.093547 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:54.093553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:54.093571 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:54.093934 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:54.094170 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:54.094172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:54.094173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:54.094314 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:54.094645 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:54.094650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:54.094898 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.095080 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.095353 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260927 00:38:54.097277 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36239
I20260927 00:38:54.097561 1421 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:54.098069 1421 heartbeater.cc:511] Master 127.0.98.62:36239 requested a full tablet report, sending...
I20260927 00:38:54.099089 959 ts_manager.cc:194] Registered new tserver with Master: e0a874e97b01410ab228958653e95ae4 (127.0.98.3:44263)
I20260927 00:38:54.099638 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41069
I20260927 00:38:54.105773 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:38:54.116087 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.109252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.0.98.62:36239)
0927 00:38:54.109398 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.109402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.109509 (+ 107us) server_negotiation.cc:407] Connection header received
0927 00:38:54.109626 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.109629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.109674 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.109739 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:54.110429 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.110986 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.111794 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.112001 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.112087 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.113394 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:54.113397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:54.113399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:54.113424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:54.114453 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.114905 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.114909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.114911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.114986 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.115217 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.115220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.115222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.115447 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:54.115576 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.115815 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.115931 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:38:54.121112 959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:38:54.124189 959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:54.140486 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.134048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44263 (local address 127.0.0.1:56686)
0927 00:38:54.135258 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.135272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.135421 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.136151 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.136154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.136185 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:54.136337 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.136344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137807 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.137811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.138985 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.138993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.139849 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.139982 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.140027 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.140137 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.140311 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":3,"spinlock_wait_cycles":7680,"thread_start_us":132,"threads_started":1}
I20260927 00:38:54.140486 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.134206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.98.3:44263)
0927 00:38:54.135129 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.135141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.135350 (+ 209us) server_negotiation.cc:407] Connection header received
0927 00:38:54.135613 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.135619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.135841 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.135988 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:54.136459 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137675 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139132 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.139716 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139822 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.139988 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.140182 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.140317 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":79,"threads_started":1}
I20260927 00:38:54.140854 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.134667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32951 (local address 127.0.0.1:47292)
0927 00:38:54.135438 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.135450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.135524 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.135987 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.135993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.136017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:54.136156 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.136163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137682 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.137686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.138916 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.138928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.140102 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.140106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.140482 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.140526 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.140612 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.140695 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":69,"thread_start_us":94,"threads_started":1}
I20260927 00:38:54.141394 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.134886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.0.98.2:32951)
0927 00:38:54.135286 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.135298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.135607 (+ 309us) server_negotiation.cc:407] Connection header received
0927 00:38:54.135672 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.135676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.135827 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.135967 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:54.136297 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137526 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139045 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.139801 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139888 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.140036 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.141122 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.141237 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":113,"threads_started":1}
I20260927 00:38:54.141584 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.134256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.0.98.1:46473)
0927 00:38:54.135446 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.135461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.135549 (+ 88us) server_negotiation.cc:407] Connection header received
0927 00:38:54.135618 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.135622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.135841 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.135988 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:54.136486 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137226 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.138890 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.139756 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.139886 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.140036 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.140931 (+ 895us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.141423 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":180,"threads_started":1}
I20260927 00:38:54.140854 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.133971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46473 (local address 127.0.0.1:60010)
0927 00:38:54.135043 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.135102 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.135226 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.136151 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.136154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.136185 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:54.136337 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.136344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.137364 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.137367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.138757 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.138767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.140102 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.140106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.140486 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.140614 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.140675 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.140732 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":193,"spinlock_wait_cycles":2816,"thread_start_us":107,"threads_started":1}
I20260927 00:38:54.144044 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1eeb3300464431a1f1550eb113b1cd (DEFAULT_TABLE table=test-table [id=10781de86f0b47389db0acc33ddcfa8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:54.144104 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1eeb3300464431a1f1550eb113b1cd (DEFAULT_TABLE table=test-table [id=10781de86f0b47389db0acc33ddcfa8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:54.145116 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1eeb3300464431a1f1550eb113b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:54.145190 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1eeb3300464431a1f1550eb113b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:54.145162 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1eeb3300464431a1f1550eb113b1cd (DEFAULT_TABLE table=test-table [id=10781de86f0b47389db0acc33ddcfa8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:54.146190 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1eeb3300464431a1f1550eb113b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:54.151675 1438 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Bootstrap starting.
I20260927 00:38:54.152797 1439 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Bootstrap starting.
I20260927 00:38:54.154242 1438 tablet_bootstrap.cc:654] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:54.154448 1440 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Bootstrap starting.
I20260927 00:38:54.155259 1439 tablet_bootstrap.cc:654] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:54.155740 1438 log.cc:824] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:54.156509 1440 tablet_bootstrap.cc:654] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:54.156695 1439 log.cc:824] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:54.157939 1440 log.cc:824] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:54.158658 1439 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: No bootstrap required, opened a new log
I20260927 00:38:54.158785 1438 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: No bootstrap required, opened a new log
I20260927 00:38:54.158874 1439 ts_tablet_manager.cc:1397] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:38:54.158982 1438 ts_tablet_manager.cc:1397] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260927 00:38:54.160156 1440 tablet_bootstrap.cc:492] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: No bootstrap required, opened a new log
I20260927 00:38:54.160386 1440 ts_tablet_manager.cc:1397] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 00:38:54.162045 1439 raft_consensus.cc:348] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.162329 1439 raft_consensus.cc:374] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:54.162385 1439 raft_consensus.cc:729] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fd52b2df584fe78249250db9ca1746, State: Initialized, Role: FOLLOWER
I20260927 00:38:54.162987 1439 consensus_queue.cc:260] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.162950 1438 raft_consensus.cc:348] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.163235 1438 raft_consensus.cc:374] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:54.163293 1438 raft_consensus.cc:729] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a874e97b01410ab228958653e95ae4, State: Initialized, Role: FOLLOWER
I20260927 00:38:54.163712 1287 heartbeater.cc:503] Master 127.0.98.62:36239 was elected leader, sending a full tablet report...
I20260927 00:38:54.163769 1438 consensus_queue.cc:260] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.164054 1439 ts_tablet_manager.cc:1428] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:38:54.164816 1438 ts_tablet_manager.cc:1428] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 00:38:54.165163 1440 raft_consensus.cc:348] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.165472 1440 raft_consensus.cc:374] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:54.165529 1440 raft_consensus.cc:729] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad6793c82ea40ceafc9e0d13b6da77c, State: Initialized, Role: FOLLOWER
I20260927 00:38:54.165922 1440 consensus_queue.cc:260] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.166692 1153 heartbeater.cc:503] Master 127.0.98.62:36239 was elected leader, sending a full tablet report...
I20260927 00:38:54.166849 1440 ts_tablet_manager.cc:1428] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 00:38:54.167012 1421 heartbeater.cc:503] Master 127.0.98.62:36239 was elected leader, sending a full tablet report...
W20260927 00:38:54.332405 1422 tablet.cc:2369] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:38:54.366545 1288 tablet.cc:2369] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:38:54.391357 1154 tablet.cc:2369] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:54.469169 1444 raft_consensus.cc:482] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:54.469388 1444 raft_consensus.cc:504] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.470623 1444 leader_election.cc:288] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad6793c82ea40ceafc9e0d13b6da77c (127.0.98.1:46473), e0a874e97b01410ab228958653e95ae4 (127.0.98.3:44263)
I20260927 00:38:54.476394 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.471015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35009 (local address 127.0.98.1:46473)
0927 00:38:54.471147 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.471153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.471432 (+ 279us) server_negotiation.cc:407] Connection header received
0927 00:38:54.471531 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.471534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.471592 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.471692 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:54.472282 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.473129 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475027 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.475663 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475799 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.475876 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.476140 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.476268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:38:54.476394 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.471108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44231 (local address 127.0.98.3:44263)
0927 00:38:54.471241 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.471245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.471428 (+ 183us) server_negotiation.cc:407] Connection header received
0927 00:38:54.471531 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.471534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.471641 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.471716 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:54.472267 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.473103 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475026 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.475733 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475827 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.475877 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.476140 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.476249 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:38:54.476440 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.470925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44263 (local address 127.0.98.2:44231)
0927 00:38:54.471298 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.471308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.471431 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.471777 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.471781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.471813 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:54.472090 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.472098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.473283 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.473287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.474828 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.474844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.475829 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.475955 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.476033 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.476112 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.476246 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":197,"spinlock_wait_cycles":35712,"thread_start_us":116,"threads_started":1}
I20260927 00:38:54.476442 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.470886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46473 (local address 127.0.98.2:35009)
0927 00:38:54.471253 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:54.471268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:54.471431 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:54.471777 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:54.471781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:54.471813 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:54.472074 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.472085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.473283 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.473287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.474827 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:54.474844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.475816 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.475820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:54.475979 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.476032 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 00:38:54.476112 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:38:54.476247 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":225,"spinlock_wait_cycles":12544,"thread_start_us":98,"threads_started":1}
I20260927 00:38:54.477177 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1eeb3300464431a1f1550eb113b1cd" candidate_uuid: "62fd52b2df584fe78249250db9ca1746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a874e97b01410ab228958653e95ae4" is_pre_election: true
I20260927 00:38:54.477496 1375 raft_consensus.cc:2454] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62fd52b2df584fe78249250db9ca1746 in term 0.
I20260927 00:38:54.477406 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1eeb3300464431a1f1550eb113b1cd" candidate_uuid: "62fd52b2df584fe78249250db9ca1746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" is_pre_election: true
I20260927 00:38:54.477800 1107 raft_consensus.cc:2454] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62fd52b2df584fe78249250db9ca1746 in term 0.
I20260927 00:38:54.478384 1176 leader_election.cc:302] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fd52b2df584fe78249250db9ca1746, 9ad6793c82ea40ceafc9e0d13b6da77c; no voters:
I20260927 00:38:54.478683 1444 raft_consensus.cc:2781] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:54.478765 1444 raft_consensus.cc:482] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:54.478796 1444 raft_consensus.cc:3037] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:54.479893 1444 raft_consensus.cc:504] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.480314 1444 leader_election.cc:288] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad6793c82ea40ceafc9e0d13b6da77c (127.0.98.1:46473), e0a874e97b01410ab228958653e95ae4 (127.0.98.3:44263)
I20260927 00:38:54.480711 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1eeb3300464431a1f1550eb113b1cd" candidate_uuid: "62fd52b2df584fe78249250db9ca1746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a874e97b01410ab228958653e95ae4"
I20260927 00:38:54.480736 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1eeb3300464431a1f1550eb113b1cd" candidate_uuid: "62fd52b2df584fe78249250db9ca1746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c"
I20260927 00:38:54.480839 1375 raft_consensus.cc:3037] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:54.480854 1107 raft_consensus.cc:3037] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:54.481879 1107 raft_consensus.cc:2454] T 8c1eeb3300464431a1f1550eb113b1cd P 9ad6793c82ea40ceafc9e0d13b6da77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62fd52b2df584fe78249250db9ca1746 in term 1.
I20260927 00:38:54.481972 1375 raft_consensus.cc:2454] T 8c1eeb3300464431a1f1550eb113b1cd P e0a874e97b01410ab228958653e95ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62fd52b2df584fe78249250db9ca1746 in term 1.
I20260927 00:38:54.482214 1176 leader_election.cc:302] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fd52b2df584fe78249250db9ca1746, 9ad6793c82ea40ceafc9e0d13b6da77c; no voters:
I20260927 00:38:54.482378 1444 raft_consensus.cc:2781] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:54.482610 1444 raft_consensus.cc:686] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [term 1 LEADER]: Becoming Leader. State: Replica: 62fd52b2df584fe78249250db9ca1746, State: Running, Role: LEADER
I20260927 00:38:54.482964 1444 consensus_queue.cc:237] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } }
I20260927 00:38:54.486058 962 catalog_manager.cc:5495] T 8c1eeb3300464431a1f1550eb113b1cd P 62fd52b2df584fe78249250db9ca1746 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62fd52b2df584fe78249250db9ca1746 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "62fd52b2df584fe78249250db9ca1746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad6793c82ea40ceafc9e0d13b6da77c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a874e97b01410ab228958653e95ae4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fd52b2df584fe78249250db9ca1746" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32951 } health_report { overall_health: HEALTHY } } }
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:54.519471 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.507839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.0.98.1:46473)
0927 00:38:54.508052 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.508081 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.508339 (+ 258us) server_negotiation.cc:407] Connection header received
0927 00:38:54.508499 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.508501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.508555 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.508649 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:54.509500 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.510036 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.511106 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.511378 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.511491 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.514979 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:54.514994 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:54.515013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:54.515049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:54.517267 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.517976 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.517983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.517984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.518155 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.518484 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.518486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.518487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.518763 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:54.518929 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.519164 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.519284 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:54.552695 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.541769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.0.98.1:46473)
0927 00:38:54.542068 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.542073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.542094 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:38:54.542336 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.542340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.542394 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.542499 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:54.543402 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.543959 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.545141 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.545411 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.545511 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.548624 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:54.548627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:54.548630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:54.548666 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:54.550275 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.550968 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.550972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.550973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.551066 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.551419 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.551422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.551423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.551638 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:54.551756 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.552366 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.552538 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:54.583871 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:54.573679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.0.98.1:46473)
0927 00:38:54.573903 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:54.573908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:54.574034 (+ 126us) server_negotiation.cc:407] Connection header received
0927 00:38:54.574213 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:54.574217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:54.574269 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:54.574402 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:54.575267 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.575854 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.577192 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:54.577456 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:54.577573 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:54.580318 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:54.580322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:54.580325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:54.580384 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:54.582056 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.582588 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.582592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.582593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.582680 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:54.582949 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:54.582953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:54.582955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:54.583171 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:54.583304 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:54.583542 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:38:54.583686 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 00:38:54.585772 1067 tablet_service.cc:2908] Rejecting scan request for tablet 8c1eeb3300464431a1f1550eb113b1cd: Uninitialized: clean time has not yet been initialized
I20260927 00:38:54.588958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1024
I20260927 00:38:54.595762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1158
I20260927 00:38:54.601806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1292
I20260927 00:38:54.607365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 930
2026-09-27T00:38:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2547 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[1463](info): set up 2 sockets
Sep 27 00:38:54 dist-test-slave-xltq krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 00:38:55.596104 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:38:55 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:38:55.720765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:55.835835 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:55.836143 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:55.836220 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:55.840500 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:55.840583 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:38:55.840600 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:55.840616 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:55.840631 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:55.840646 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:55.846192 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46331
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:55.847334 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:55.848387 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:55.857879 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:55.857879 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:55.857920 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:55.858064 1476 server_base.cc:1034] running on GCE node
I20260927 00:38:55.858841 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:55.860018 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:55.861179 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790469535861153 us; error 42 us; skew 500 ppm
Sep 27 00:38:55 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:55.864060 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:38:55.865021 1476 webserver.cc:460] Webserver started at http://127.0.98.62:37731/ using document root <none> and password file <none>
I20260927 00:38:55.865586 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:55.865659 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:55.865896 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:55.868068 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "ab85ea060f974bbf919b9bd6deefa7c4"
format_stamp: "Formatted at 2026-09-27 00:38:55 on dist-test-slave-xltq"
server_key: "5552e88fdaaf1e13c7179881d2774dd8"
server_key_iv: "62d14e130a61181536841fad9277e8ea"
server_key_version: "encryptionkey@0"
I20260927 00:38:55.868714 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab85ea060f974bbf919b9bd6deefa7c4"
format_stamp: "Formatted at 2026-09-27 00:38:55 on dist-test-slave-xltq"
server_key: "5552e88fdaaf1e13c7179881d2774dd8"
server_key_iv: "62d14e130a61181536841fad9277e8ea"
server_key_version: "encryptionkey@0"
I20260927 00:38:55.872601 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:38:55.875245 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:55.876776 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:55.876907 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ab85ea060f974bbf919b9bd6deefa7c4"
format_stamp: "Formatted at 2026-09-27 00:38:55 on dist-test-slave-xltq"
server_key: "5552e88fdaaf1e13c7179881d2774dd8"
server_key_iv: "62d14e130a61181536841fad9277e8ea"
server_key_version: "encryptionkey@0"
I20260927 00:38:55.877012 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:55.909467 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:55.913954 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:55.914129 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:55.922037 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46331
I20260927 00:38:55.922056 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46331 every 8 connection(s)
I20260927 00:38:55.923108 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:55.926213 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:55.927918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1476
I20260927 00:38:55.928011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:38:55.928284 392 external_mini_cluster.cc:1428] Setting key 7f78c2a5f0853439ed3db2abf85d67f2
I20260927 00:38:55.933071 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Bootstrap starting.
I20260927 00:38:55.935734 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:55.936522 1554 log.cc:824] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:38:55 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:55.938761 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: No bootstrap required, opened a new log
I20260927 00:38:55.942592 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 00:38:55.942836 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:55.942873 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab85ea060f974bbf919b9bd6deefa7c4, State: Initialized, Role: FOLLOWER
I20260927 00:38:55.943055 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:55.929711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.62:46331)
0927 00:38:55.930171 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:55.930193 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:55.930237 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:38:55.931088 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:55.931125 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:55.931574 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:55.931963 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:55.932635 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:55.934051 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:55.934952 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:55.935661 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:55.935829 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:55.938440 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:55.938446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:55.938460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:55.938486 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:55.940086 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:55.940629 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:55.940635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:55.940640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:55.940760 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:55.941035 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:55.941037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:55.941038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:55.941472 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:55.941630 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:55.941906 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 00:38:55.942196 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":131,"threads_started":1}
I20260927 00:38:55.943382 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 00:38:55.943545 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:55.943615 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:55.943712 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:55.944753 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 00:38:55.945091 1554 leader_election.cc:302] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab85ea060f974bbf919b9bd6deefa7c4; no voters:
I20260927 00:38:55.945438 1554 leader_election.cc:288] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:55.945540 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:55.945730 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [term 1 LEADER]: Becoming Leader. State: Replica: ab85ea060f974bbf919b9bd6deefa7c4, State: Running, Role: LEADER
I20260927 00:38:55.946026 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } }
I20260927 00:38:55.946614 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:55.947609 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab85ea060f974bbf919b9bd6deefa7c4. Latest consensus state: current_term: 1 leader_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 00:38:55.947652 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab85ea060f974bbf919b9bd6deefa7c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46331 } } }
I20260927 00:38:55.947736 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:55.948125 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:55.947736 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:55.951341 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:55.957599 1568 catalog_manager.cc:1347] Generated new cluster ID: 829ff296aa884760addad1feb255aa7e
I20260927 00:38:55.957679 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:55.974623 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:55.975960 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:55.985054 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Generated new TSK 0
I20260927 00:38:55.985999 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:38:56.019109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:56.136406 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:56.136721 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:56.136770 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:56.141002 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:56.141100 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:56.141144 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:56.141180 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:38:56.146835 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:56.148015 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:56.149118 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:56.159272 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.159324 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.159286 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:56.159607 1580 server_base.cc:1034] running on GCE node
I20260927 00:38:56.160264 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:56.160936 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:56.162134 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790469536162122 us; error 45 us; skew 500 ppm
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:56.165287 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:38:56.166309 1580 webserver.cc:460] Webserver started at http://127.0.98.1:37687/ using document root <none> and password file <none>
I20260927 00:38:56.166891 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:56.166973 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:56.167196 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:56.169428 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "ac4f894b28cd4f81b99dee05d36512a7"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "cefb7901f22569f9b59f169943faa2bf"
server_key_iv: "da8548ef5cd36369d2f3ff04e57505f3"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.170025 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ac4f894b28cd4f81b99dee05d36512a7"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "cefb7901f22569f9b59f169943faa2bf"
server_key_iv: "da8548ef5cd36369d2f3ff04e57505f3"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.174243 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:56.176983 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.178463 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:38:56.178603 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "ac4f894b28cd4f81b99dee05d36512a7"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "cefb7901f22569f9b59f169943faa2bf"
server_key_iv: "da8548ef5cd36369d2f3ff04e57505f3"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.178726 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:56.190804 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:56.194201 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:56.194386 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:56.195932 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:56.195986 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.196039 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:56.196066 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:56.206681 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46001
I20260927 00:38:56.206842 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46001 every 8 connection(s)
I20260927 00:38:56.207787 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:38:56.210858 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.196824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38685 (local address 127.0.98.62:46331)
0927 00:38:56.196982 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.196988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.197942 (+ 954us) server_negotiation.cc:407] Connection header received
0927 00:38:56.199245 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.199249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.199311 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.199404 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.201149 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.201722 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.203553 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.203936 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.204027 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.206807 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.206811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.206814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.206843 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.208638 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.209206 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.209211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.209213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.209306 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.209657 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.209661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.209663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.209878 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.209977 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.210504 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.210671 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:38:56.211687 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.197230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:38685)
0927 00:38:56.197753 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.197817 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.198940 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.199642 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.199653 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.200176 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.200971 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.200998 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.201849 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.201854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.203429 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.203442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.204112 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.204115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.204350 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.205052 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.205069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.206630 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.208771 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.208778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.208795 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.209095 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.209399 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.209402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.209404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.209551 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.209985 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.209992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.210258 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.210444 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.210830 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260927 00:38:56.213096 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 00:38:56.213260 1711 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:56.213822 1711 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 00:38:56.215624 1508 ts_manager.cc:194] Registered new tserver with Master: ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1:46001)
I20260927 00:38:56.215657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1580
I20260927 00:38:56.215776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:38:56.216035 392 external_mini_cluster.cc:1428] Setting key e4d1532bd80f43d39fb53cb369d08895
I20260927 00:38:56.217026 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38685
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:38:56.225026 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.215633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47667 (local address 127.0.98.62:46331)
0927 00:38:56.217671 (+ 2038us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.217675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.217690 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:56.217725 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.217728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.217767 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.217856 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.218391 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.219146 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.220338 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.220570 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.220661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.221820 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.221824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.221827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.221851 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.223405 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.223830 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.223834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.223836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.223914 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.224172 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.224175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.224177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.224408 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.224496 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.224754 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.224865 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1971}
I20260927 00:38:56.225049 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.215544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.1:47667)
0927 00:38:56.216410 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.216425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.216522 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.217907 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.217911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.218127 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.218264 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.218271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.219326 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.219330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.220147 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.220157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.220676 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.220679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.220766 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.221254 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.221267 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.221679 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.223531 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.223535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.223537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.223736 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.223997 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.223999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.224002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.224095 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.224500 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.224503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.224599 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.224685 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.224852 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":222,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:38:56.244004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:56.354701 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:56.354986 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:56.355031 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:56.359061 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:56.359129 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:56.359169 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:56.359200 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:38:56.364650 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:56.365718 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:56.366779 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:56.376482 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.376483 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.376482 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:56.376850 1720 server_base.cc:1034] running on GCE node
I20260927 00:38:56.377317 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:56.377969 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:56.379158 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790469536379131 us; error 37 us; skew 500 ppm
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:56.382166 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:38:56.383167 1720 webserver.cc:460] Webserver started at http://127.0.98.2:45317/ using document root <none> and password file <none>
I20260927 00:38:56.383759 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:56.383837 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:56.384057 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:56.386176 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "83a77807fccd4a1bb4d3a2d419599d05"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "8b00e9d566410e83d9a5cafd7e37daa0"
server_key_iv: "83e4a54f8a88d4860c4f58a26d28a546"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.386735 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83a77807fccd4a1bb4d3a2d419599d05"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "8b00e9d566410e83d9a5cafd7e37daa0"
server_key_iv: "83e4a54f8a88d4860c4f58a26d28a546"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.390641 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:56.393246 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.394579 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:56.394708 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "83a77807fccd4a1bb4d3a2d419599d05"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "8b00e9d566410e83d9a5cafd7e37daa0"
server_key_iv: "83e4a54f8a88d4860c4f58a26d28a546"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.394814 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:56.416939 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:56.420130 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:56.420351 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:56.421802 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:56.421860 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.421914 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:56.421941 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.431983 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35615
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:56.432050 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35615 every 8 connection(s)
I20260927 00:38:56.433061 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:38:56.436632 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.422679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53527 (local address 127.0.98.62:46331)
0927 00:38:56.422790 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.422794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.423938 (+ 1144us) server_negotiation.cc:407] Connection header received
0927 00:38:56.425015 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.425018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.425075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.425159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.426833 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.427645 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.429426 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.429649 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.429738 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.432803 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.432808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.432810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.432841 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.434320 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.435044 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.435048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.435049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.435138 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.435542 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.435545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.435546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.435763 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.435846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.436372 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.436492 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:38:56.437476 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.423273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:53527)
0927 00:38:56.423754 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.423797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.424767 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.425435 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.425447 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.425916 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.426649 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.426670 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.427798 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.427802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.429287 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.429298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.429811 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.429817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.430046 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.430794 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.430816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.432602 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.434444 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.434448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.434462 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.434893 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.435256 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.435258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.435260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.435403 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.435843 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.435848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.436088 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.436283 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.436566 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":133,"threads_started":1}
I20260927 00:38:56.438825 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 00:38:56.438979 1851 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:56.439572 1851 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 00:38:56.440812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1720
I20260927 00:38:56.440935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:38:56.440915 1508 ts_manager.cc:194] Registered new tserver with Master: 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615)
I20260927 00:38:56.441286 392 external_mini_cluster.cc:1428] Setting key a12ac3ff4c6b24a9f38fe0d7541df08a
I20260927 00:38:56.441637 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53527
I20260927 00:38:56.449301 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.441337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57755 (local address 127.0.98.62:46331)
0927 00:38:56.441569 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.441573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.441796 (+ 223us) server_negotiation.cc:407] Connection header received
0927 00:38:56.441960 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.441963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.442013 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.442085 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.442619 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.443357 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.444442 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.444666 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.444757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.445927 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.445930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.445932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.445954 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.447571 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.448106 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.448109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.448110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.448186 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.448474 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.448476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.448477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.448702 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.448799 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.449077 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.449177 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260927 00:38:56.449519 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.441277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.2:57755)
0927 00:38:56.441724 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.441739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.441863 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.442109 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.442112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.442371 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.442489 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.442497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.443500 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.443504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.444327 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.444339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.444765 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.444769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.444862 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.445340 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.445355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.445790 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.447706 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.447710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.447713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.448011 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.448300 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.448304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.448306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.448401 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.448802 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.448805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.448910 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.449031 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.449203 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:38:56.468359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46331
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:56.584499 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:56.584793 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:56.584839 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:56.589043 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:56.589121 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:56.589162 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:56.589196 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:38:56.594713 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:56.595793 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:56.596843 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:56.606771 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.606813 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:56.606771 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:56.606971 1860 server_base.cc:1034] running on GCE node
I20260927 00:38:56.607518 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:56.608186 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:56.609400 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790469536609364 us; error 49 us; skew 500 ppm
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:56.612373 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:38:56.613464 1860 webserver.cc:460] Webserver started at http://127.0.98.3:40941/ using document root <none> and password file <none>
I20260927 00:38:56.614111 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:56.614200 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:56.614415 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:56.616727 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d86f4ca5eae4164a009306a75e87c8e"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "c8a02a5aaa14e1246d0119c58fa35433"
server_key_iv: "227d83f064c19ed973bde2dfc2c30b77"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.617326 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d86f4ca5eae4164a009306a75e87c8e"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "c8a02a5aaa14e1246d0119c58fa35433"
server_key_iv: "227d83f064c19ed973bde2dfc2c30b77"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.621472 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:38:56.624429 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.625838 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:38:56.625982 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "9d86f4ca5eae4164a009306a75e87c8e"
format_stamp: "Formatted at 2026-09-27 00:38:56 on dist-test-slave-xltq"
server_key: "c8a02a5aaa14e1246d0119c58fa35433"
server_key_iv: "227d83f064c19ed973bde2dfc2c30b77"
server_key_version: "encryptionkey@0"
I20260927 00:38:56.626096 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:56.650606 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:56.654024 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:56.654213 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:56.655907 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:56.655962 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:56.656018 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:56.656046 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:56.666441 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36541
I20260927 00:38:56.666474 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36541 every 8 connection(s)
I20260927 00:38:56.667510 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:38:56.669865 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.656883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52021 (local address 127.0.98.62:46331)
0927 00:38:56.657000 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.657004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.658239 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 00:38:56.659417 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.659422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.659485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.659587 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.660963 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.661490 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.662794 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.663082 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.663184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.666228 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.666233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.666235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.666268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.667764 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.668388 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.668392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.668394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.668492 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.668824 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.668826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.668828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.669015 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.669089 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.669600 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.669719 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:38:56.670615 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.657471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:52021)
0927 00:38:56.658021 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.658071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.659175 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.659775 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.659785 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.660222 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.660790 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.660807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.661616 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.661619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.662666 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.662679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.663260 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.663264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.663422 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.664097 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.664115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.666017 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.667895 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.667902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.667919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.668290 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.668576 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.668578 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.668579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.668740 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.669090 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.669095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.669328 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.669480 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.669736 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":134,"threads_started":1}
I20260927 00:38:56.671826 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46331
I20260927 00:38:56.671979 1991 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:56.672587 1991 heartbeater.cc:511] Master 127.0.98.62:46331 requested a full tablet report, sending...
I20260927 00:38:56.673822 1507 ts_manager.cc:194] Registered new tserver with Master: 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541)
I20260927 00:38:56.674512 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52021
I20260927 00:38:56.675283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 1860
I20260927 00:38:56.675396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:38:56.675676 392 external_mini_cluster.cc:1428] Setting key e28a0070803ecb0e472b33efa5897e19
I20260927 00:38:56.677896 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:38:56.682111 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.674332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47081 (local address 127.0.98.62:46331)
0927 00:38:56.674461 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.674466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.674707 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:38:56.674834 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.674837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.674882 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.674943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.675513 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.676077 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.677226 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.677442 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.677529 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.678819 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.678824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.678827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.678855 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.680364 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.680807 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.680811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.680813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.680899 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.681170 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.681173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.681176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.681407 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.681503 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.681820 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.681935 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:38:56.682142 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.674219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.98.3:47081)
0927 00:38:56.674602 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.674618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.674734 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.674979 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.674982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.675227 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.675340 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.675348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.676273 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.676277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.677056 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.677066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.677556 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.677560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.677676 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.678288 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.678304 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.678673 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.680470 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.680473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.680476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.680701 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.680987 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.680990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.680992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.681081 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.681513 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.681516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.681651 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.681751 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.681924 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":108,"threads_started":1}
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:56.705868 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.696855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.98.62:46331)
0927 00:38:56.697009 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.697013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.697028 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:56.697069 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.697072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.697125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.697203 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.697977 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.698518 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.699455 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.699677 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.699769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.702310 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.702313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.702316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.702347 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.704064 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.704641 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.704646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.704648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.704743 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.705009 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.705012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.705013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.705206 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.705341 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.705588 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.705715 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:38:56.709472 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:38:56.713152 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:56.729637 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.723507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.98.1:46001)
0927 00:38:56.723847 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.723855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.724523 (+ 668us) server_negotiation.cc:407] Connection header received
0927 00:38:56.724610 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.724615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.724853 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.725014 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.725692 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.726549 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.728091 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.728903 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.729019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.729191 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.729375 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.729489 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260927 00:38:56.729631 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.723356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46001 (local address 127.0.0.1:51748)
0927 00:38:56.724130 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.724190 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.724374 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.725182 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.725188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.725208 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.725374 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.725382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.726668 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.726671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.727908 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.727917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.729026 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.729030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.729156 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.729205 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.729327 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.729463 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":100,"threads_started":1}
I20260927 00:38:56.730530 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.724265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36541 (local address 127.0.0.1:45904)
0927 00:38:56.724942 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.724955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.725042 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.725710 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.725713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.725731 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.725839 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.725844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.727216 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.727219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.728438 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.728449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.729691 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.729695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.729855 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.729878 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.730303 (+ 425us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.730390 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260927 00:38:56.730729 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.724408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.0.98.3:36541)
0927 00:38:56.725263 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.725272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.725290 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:38:56.725348 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.725351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.725525 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.725709 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.725964 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.727063 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.728577 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.729354 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.729454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.729622 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.730399 (+ 777us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.730502 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":100,"threads_started":1}
I20260927 00:38:56.730852 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.723545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.0.1:34534)
0927 00:38:56.724898 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.724912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.724993 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.725510 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.725513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.725529 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.725644 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.725683 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.727367 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.727371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.728773 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.728781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.730452 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.730456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.730561 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.730584 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.730657 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.730727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":116,"threads_started":1}
I20260927 00:38:56.731477 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.724389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.2:35615)
0927 00:38:56.724800 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.724812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.725080 (+ 268us) server_negotiation.cc:407] Connection header received
0927 00:38:56.725163 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.725168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.725357 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.725507 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.725816 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.727009 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.728896 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.729694 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.730067 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.730382 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.731195 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.731309 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":112,"threads_started":1}
I20260927 00:38:56.733924 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 7e38a259e71642189402602eaa82f9eb (DEFAULT_TABLE table=test-table [id=ef53f5c151dc4f849e9441297e8c1653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.733924 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 7e38a259e71642189402602eaa82f9eb (DEFAULT_TABLE table=test-table [id=ef53f5c151dc4f849e9441297e8c1653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.733934 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7e38a259e71642189402602eaa82f9eb (DEFAULT_TABLE table=test-table [id=ef53f5c151dc4f849e9441297e8c1653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.735023 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e38a259e71642189402602eaa82f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.735016 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e38a259e71642189402602eaa82f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.735013 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e38a259e71642189402602eaa82f9eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.742602 2014 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Bootstrap starting.
I20260927 00:38:56.743618 2016 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Bootstrap starting.
I20260927 00:38:56.743829 2015 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Bootstrap starting.
I20260927 00:38:56.745159 2014 tablet_bootstrap.cc:654] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.745878 2016 tablet_bootstrap.cc:654] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.745878 2015 tablet_bootstrap.cc:654] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.745908 2014 log.cc:824] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:56.746704 2016 log.cc:824] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:56.747284 2015 log.cc:824] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:56.748310 2014 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: No bootstrap required, opened a new log
I20260927 00:38:56.748535 2014 ts_tablet_manager.cc:1397] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:38:56.748883 2016 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: No bootstrap required, opened a new log
I20260927 00:38:56.749061 2016 ts_tablet_manager.cc:1397] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:38:56.749320 2015 tablet_bootstrap.cc:492] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: No bootstrap required, opened a new log
I20260927 00:38:56.749511 2015 ts_tablet_manager.cc:1397] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:38:56.752485 2016 raft_consensus.cc:348] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.752799 2016 raft_consensus.cc:374] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.752868 2016 raft_consensus.cc:729] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d86f4ca5eae4164a009306a75e87c8e, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.753516 2016 consensus_queue.cc:260] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.753396 2014 raft_consensus.cc:348] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.753702 2014 raft_consensus.cc:374] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.753755 2014 raft_consensus.cc:729] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a77807fccd4a1bb4d3a2d419599d05, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.754426 2014 consensus_queue.cc:260] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.755080 2016 ts_tablet_manager.cc:1428] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:38:56.754839 2015 raft_consensus.cc:348] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.755260 2015 raft_consensus.cc:374] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.755334 2015 raft_consensus.cc:729] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4f894b28cd4f81b99dee05d36512a7, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.755512 1991 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 00:38:56.755844 2015 consensus_queue.cc:260] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.756031 1851 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 00:38:56.756973 2015 ts_tablet_manager.cc:1428] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:38:56.757006 2014 ts_tablet_manager.cc:1428] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:38:56.757328 1711 heartbeater.cc:503] Master 127.0.98.62:46331 was elected leader, sending a full tablet report...
I20260927 00:38:56.778117 2022 raft_consensus.cc:482] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:56.778349 2022 raft_consensus.cc:504] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.779480 2022 leader_election.cc:288] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541), 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615)
I20260927 00:38:56.783782 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.779739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.98.1:58039)
0927 00:38:56.780164 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.780179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.780360 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.780629 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.780632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.780649 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.780783 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.780789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.781589 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.781592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.782609 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.782619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.783361 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.783365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.783460 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.783478 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.783549 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.783630 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":38,"thread_start_us":166,"threads_started":1}
I20260927 00:38:56.783836 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.779964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58039 (local address 127.0.98.2:35615)
0927 00:38:56.780095 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.780099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.780273 (+ 174us) server_negotiation.cc:407] Connection header received
0927 00:38:56.780480 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.780483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.780537 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.780600 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.780902 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.781459 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.782737 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.783232 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.783341 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.783405 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.783614 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.783709 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:38:56.783836 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.780049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60753 (local address 127.0.98.3:36541)
0927 00:38:56.780190 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.780195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.780425 (+ 230us) server_negotiation.cc:407] Connection header received
0927 00:38:56.780552 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.780570 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.780628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.780702 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.780990 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.781513 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.782726 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.783135 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.783260 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.783334 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.783647 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.783725 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:38:56.784291 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.779818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36541 (local address 127.0.98.1:60753)
0927 00:38:56.780326 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.780339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.780445 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.780743 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.780745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.780767 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.780872 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.780879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.781628 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.781632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.782602 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.782612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.783291 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.783295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.783417 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.783442 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.784037 (+ 595us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.784114 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":37,"thread_start_us":88,"threads_started":1}
I20260927 00:38:56.784533 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e38a259e71642189402602eaa82f9eb" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05" is_pre_election: true
I20260927 00:38:56.784876 1805 raft_consensus.cc:2454] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 0.
I20260927 00:38:56.784905 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e38a259e71642189402602eaa82f9eb" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d86f4ca5eae4164a009306a75e87c8e" is_pre_election: true
I20260927 00:38:56.785190 1945 raft_consensus.cc:2454] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 0.
I20260927 00:38:56.785470 1598 leader_election.cc:302] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a77807fccd4a1bb4d3a2d419599d05, ac4f894b28cd4f81b99dee05d36512a7; no voters:
I20260927 00:38:56.785841 2022 raft_consensus.cc:2781] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:56.785923 2022 raft_consensus.cc:482] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:56.785957 2022 raft_consensus.cc:3037] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.787067 2022 raft_consensus.cc:504] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.787456 2022 leader_election.cc:288] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 election: Requested vote from peers 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541), 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615)
I20260927 00:38:56.787822 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e38a259e71642189402602eaa82f9eb" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d86f4ca5eae4164a009306a75e87c8e"
I20260927 00:38:56.787899 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e38a259e71642189402602eaa82f9eb" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05"
I20260927 00:38:56.787935 1945 raft_consensus.cc:3037] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.788054 1805 raft_consensus.cc:3037] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.788998 1945 raft_consensus.cc:2454] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 1.
I20260927 00:38:56.789126 1805 raft_consensus.cc:2454] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 1.
I20260927 00:38:56.789340 1597 leader_election.cc:302] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d86f4ca5eae4164a009306a75e87c8e, ac4f894b28cd4f81b99dee05d36512a7; no voters:
I20260927 00:38:56.789500 2022 raft_consensus.cc:2781] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:56.789736 2022 raft_consensus.cc:686] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 1 LEADER]: Becoming Leader. State: Replica: ac4f894b28cd4f81b99dee05d36512a7, State: Running, Role: LEADER
I20260927 00:38:56.790061 2022 consensus_queue.cc:237] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } }
I20260927 00:38:56.793475 1502 catalog_manager.cc:5495] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20260927 00:38:56.817628 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.813487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.0.98.1:46001)
0927 00:38:56.813651 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.813654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.813670 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:38:56.813937 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.813940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.814001 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.814097 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:56.814474 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.815053 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.816008 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.816276 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.816367 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.816679 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:56.817170 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:56.817259 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.817441 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.817505 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:38:56.823752 1945 raft_consensus.cc:1261] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Refusing update from remote peer ac4f894b28cd4f81b99dee05d36512a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:56.823957 1805 raft_consensus.cc:1261] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Refusing update from remote peer ac4f894b28cd4f81b99dee05d36512a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:56.824513 2025 consensus_queue.cc:1035] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:56.824733 2022 consensus_queue.cc:1035] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:56.831375 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333763222822666240 to 7333763222694727680. Current Snapshot: MvccSnapshot[applied={T|T < 7333763222822666240}]
I20260927 00:38:56.833976 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333763222822666240 to 7333763222694727680. Current Snapshot: MvccSnapshot[applied={T|T < 7333763222822666240}]
I20260927 00:38:56.845515 1502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:56.845695 1502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:56.848429 1502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Sending DeleteTablet for 3 replicas of tablet 7e38a259e71642189402602eaa82f9eb
I20260927 00:38:56.849397 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e38a259e71642189402602eaa82f9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34534
I20260927 00:38:56.849414 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e38a259e71642189402602eaa82f9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51748
I20260927 00:38:56.849946 2038 tablet_replica.cc:330] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: stopping tablet replica
I20260927 00:38:56.849972 2037 tablet_replica.cc:330] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: stopping tablet replica
I20260927 00:38:56.849974 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e38a259e71642189402602eaa82f9eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45904
I20260927 00:38:56.850337 2037 raft_consensus.cc:2229] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:38:56.850449 2038 raft_consensus.cc:2229] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:38:56.850785 2037 raft_consensus.cc:2258] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:56.850812 2038 raft_consensus.cc:2258] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:56.851184 2039 tablet_replica.cc:330] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: stopping tablet replica
I20260927 00:38:56.851486 2039 raft_consensus.cc:2229] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:38:56.851445 1502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:38:56.851805 2039 raft_consensus.cc:2258] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:38:56.851928 1502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:56.853380 2039 ts_tablet_manager.cc:1905] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:56.854733 2038 ts_tablet_manager.cc:1905] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:56.855281 2037 ts_tablet_manager.cc:1905] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:56.858081 2037 ts_tablet_manager.cc:1918] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:38:56.858194 2037 log.cc:1196] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/7e38a259e71642189402602eaa82f9eb
I20260927 00:38:56.858510 2037 ts_tablet_manager.cc:1939] T 7e38a259e71642189402602eaa82f9eb P ac4f894b28cd4f81b99dee05d36512a7: Deleting consensus metadata
I20260927 00:38:56.859797 1493 catalog_manager.cc:4841] TS ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1:46001): tablet 7e38a259e71642189402602eaa82f9eb (table test-table [id=ef53f5c151dc4f849e9441297e8c1653]) successfully deleted
I20260927 00:38:56.861196 1645 tablet_service.cc:1467] Processing CreateTablet for tablet faf21c27f8704600b0d606dab1c44849 (DEFAULT_TABLE table=test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.861462 2039 ts_tablet_manager.cc:1918] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:38:56.861508 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf21c27f8704600b0d606dab1c44849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.861466 1785 tablet_service.cc:1467] Processing CreateTablet for tablet faf21c27f8704600b0d606dab1c44849 (DEFAULT_TABLE table=test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.861560 2039 log.cc:1196] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/7e38a259e71642189402602eaa82f9eb
I20260927 00:38:56.861707 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf21c27f8704600b0d606dab1c44849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.861860 2039 ts_tablet_manager.cc:1939] T 7e38a259e71642189402602eaa82f9eb P 9d86f4ca5eae4164a009306a75e87c8e: Deleting consensus metadata
I20260927 00:38:56.863843 1493 catalog_manager.cc:4841] TS 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541): tablet 7e38a259e71642189402602eaa82f9eb (table test-table [id=ef53f5c151dc4f849e9441297e8c1653]) successfully deleted
I20260927 00:38:56.863924 2015 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Bootstrap starting.
I20260927 00:38:56.863931 2014 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Bootstrap starting.
I20260927 00:38:56.864429 1925 tablet_service.cc:1467] Processing CreateTablet for tablet faf21c27f8704600b0d606dab1c44849 (DEFAULT_TABLE table=test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:56.864701 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf21c27f8704600b0d606dab1c44849. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.864960 2014 tablet_bootstrap.cc:654] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.864979 2015 tablet_bootstrap.cc:654] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.866633 2038 ts_tablet_manager.cc:1918] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:38:56.866865 2038 log.cc:1196] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/7e38a259e71642189402602eaa82f9eb
I20260927 00:38:56.867182 2038 ts_tablet_manager.cc:1939] T 7e38a259e71642189402602eaa82f9eb P 83a77807fccd4a1bb4d3a2d419599d05: Deleting consensus metadata
I20260927 00:38:56.867291 2016 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Bootstrap starting.
I20260927 00:38:56.868115 2016 tablet_bootstrap.cc:654] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.868538 1494 catalog_manager.cc:4841] TS 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615): tablet 7e38a259e71642189402602eaa82f9eb (table test-table [id=ef53f5c151dc4f849e9441297e8c1653]) successfully deleted
I20260927 00:38:56.869439 2016 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: No bootstrap required, opened a new log
I20260927 00:38:56.869531 2015 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: No bootstrap required, opened a new log
I20260927 00:38:56.869534 2016 ts_tablet_manager.cc:1397] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:38:56.869616 2015 ts_tablet_manager.cc:1397] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:38:56.869798 2014 tablet_bootstrap.cc:492] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: No bootstrap required, opened a new log
I20260927 00:38:56.869901 2014 ts_tablet_manager.cc:1397] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:38:56.870092 2016 raft_consensus.cc:348] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.870247 2016 raft_consensus.cc:374] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.870294 2016 raft_consensus.cc:729] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d86f4ca5eae4164a009306a75e87c8e, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.870348 2015 raft_consensus.cc:348] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.870357 2014 raft_consensus.cc:348] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.870462 2015 raft_consensus.cc:374] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.870463 2014 raft_consensus.cc:374] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.870424 2016 consensus_queue.cc:260] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.870499 2014 raft_consensus.cc:729] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a77807fccd4a1bb4d3a2d419599d05, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.870499 2015 raft_consensus.cc:729] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4f894b28cd4f81b99dee05d36512a7, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.870618 2014 consensus_queue.cc:260] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.870688 2016 ts_tablet_manager.cc:1428] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:38:56.870872 2014 ts_tablet_manager.cc:1428] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:38:56.871234 2015 consensus_queue.cc:260] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.872007 2015 ts_tablet_manager.cc:1428] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:56.905418 2020 raft_consensus.cc:482] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:56.905586 2020 raft_consensus.cc:504] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.906891 2020 leader_election.cc:288] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1:46001), 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615)
I20260927 00:38:56.911125 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.907363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46377 (local address 127.0.98.2:35615)
0927 00:38:56.907515 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.907519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.907579 (+ 60us) server_negotiation.cc:407] Connection header received
0927 00:38:56.907760 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.907763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.907811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.907872 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.908172 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.908928 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.910175 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.910569 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.910663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.910747 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.910923 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.910997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:38:56.911147 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.907123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35615 (local address 127.0.98.3:46377)
0927 00:38:56.907471 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.907484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.907644 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.907938 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.907941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.907957 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.908078 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.908084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.909077 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.909080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.910028 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.910045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.910714 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.910717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.910811 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.910833 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.910899 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.910978 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
I20260927 00:38:56.912001 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.907083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46001 (local address 127.0.98.3:54339)
0927 00:38:56.907268 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.907283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.907362 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.907838 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.907841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.907856 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:56.907979 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.907985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.909045 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.909048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.910573 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.910585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.911565 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.911569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.911689 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.911719 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.911787 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.911871 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":60}
I20260927 00:38:56.912009 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.907510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54339 (local address 127.0.98.1:46001)
0927 00:38:56.907628 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.907633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.907645 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:38:56.907696 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.907699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.907742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.907818 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:56.908083 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.908917 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.910733 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.911396 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.911536 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.911604 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.911825 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.911904 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:38:56.912001 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf21c27f8704600b0d606dab1c44849" candidate_uuid: "9d86f4ca5eae4164a009306a75e87c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05" is_pre_election: true
I20260927 00:38:56.912191 1805 raft_consensus.cc:2454] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d86f4ca5eae4164a009306a75e87c8e in term 0.
I20260927 00:38:56.912647 1878 leader_election.cc:302] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a77807fccd4a1bb4d3a2d419599d05, 9d86f4ca5eae4164a009306a75e87c8e; no voters:
I20260927 00:38:56.912845 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf21c27f8704600b0d606dab1c44849" candidate_uuid: "9d86f4ca5eae4164a009306a75e87c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac4f894b28cd4f81b99dee05d36512a7" is_pre_election: true
I20260927 00:38:56.912985 2020 raft_consensus.cc:2781] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:56.913043 2020 raft_consensus.cc:482] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:56.913079 2020 raft_consensus.cc:3037] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.913125 1665 raft_consensus.cc:2454] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d86f4ca5eae4164a009306a75e87c8e in term 0.
I20260927 00:38:56.914297 2020 raft_consensus.cc:504] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.914692 2020 leader_election.cc:288] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [CANDIDATE]: Term 1 election: Requested vote from peers ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1:46001), 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615)
I20260927 00:38:56.915144 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf21c27f8704600b0d606dab1c44849" candidate_uuid: "9d86f4ca5eae4164a009306a75e87c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac4f894b28cd4f81b99dee05d36512a7"
I20260927 00:38:56.915108 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf21c27f8704600b0d606dab1c44849" candidate_uuid: "9d86f4ca5eae4164a009306a75e87c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05"
I20260927 00:38:56.915283 1665 raft_consensus.cc:3037] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.915338 1805 raft_consensus.cc:3037] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:56.916924 1665 raft_consensus.cc:2454] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d86f4ca5eae4164a009306a75e87c8e in term 1.
I20260927 00:38:56.917022 1805 raft_consensus.cc:2454] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d86f4ca5eae4164a009306a75e87c8e in term 1.
I20260927 00:38:56.917304 1877 leader_election.cc:302] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d86f4ca5eae4164a009306a75e87c8e, ac4f894b28cd4f81b99dee05d36512a7; no voters:
I20260927 00:38:56.917483 2020 raft_consensus.cc:2781] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:56.917711 2020 raft_consensus.cc:686] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 LEADER]: Becoming Leader. State: Replica: 9d86f4ca5eae4164a009306a75e87c8e, State: Running, Role: LEADER
I20260927 00:38:56.917968 2020 consensus_queue.cc:237] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
W20260927 00:38:56.918522 1992 tablet.cc:2369] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:56.920544 1508 catalog_manager.cc:5495] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9d86f4ca5eae4164a009306a75e87c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } health_report { overall_health: HEALTHY } } }
W20260927 00:38:56.934309 1852 tablet.cc:2369] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:38:56.959227 1712 tablet.cc:2369] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:38:56 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:56.967382 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.957956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46331 (local address 127.0.0.1:49000)
0927 00:38:56.958278 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:56.958294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:56.958425 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:56.958675 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:56.958678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:56.959199 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:56.959344 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.959353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.960242 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.960246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.961062 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:56.961074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.961518 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.961522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:56.961612 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.962192 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:56.962215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:56.963874 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:56.965504 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.965516 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.965524 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.965847 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.966124 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:56.966127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:56.966129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:56.966245 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:56.966714 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:56.966723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:56.966862 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 00:38:56.966991 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 00:38:56.967180 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260927 00:38:56.967506 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:56.958038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.0.98.62:46331)
0927 00:38:56.958175 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:56.958180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:56.958395 (+ 215us) server_negotiation.cc:407] Connection header received
0927 00:38:56.958511 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:56.958513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:56.958555 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:56.958633 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:56.959504 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.960083 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.961180 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:56.961398 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:56.961473 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:56.964019 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:56.964022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:56.964024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:56.964045 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:56.965381 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.965962 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.965965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.965966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.966038 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:56.966367 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:56.966377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:56.966378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:56.966593 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:56.966695 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:56.967062 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:38:56.967314 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":51}
I20260927 00:38:56.971830 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49000:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:38:56.972829 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:56.978113 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7825cdea94a02996f4095e476b9a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bd236b3a2504cacb22386eae6536ac5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:56.978113 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7825cdea94a02996f4095e476b9a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bd236b3a2504cacb22386eae6536ac5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:56.978113 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 45e7825cdea94a02996f4095e476b9a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bd236b3a2504cacb22386eae6536ac5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:56.978531 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7825cdea94a02996f4095e476b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.978531 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7825cdea94a02996f4095e476b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.978542 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45e7825cdea94a02996f4095e476b9a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:56.981475 2015 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: Bootstrap starting.
I20260927 00:38:56.981845 2014 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: Bootstrap starting.
I20260927 00:38:56.982473 2015 tablet_bootstrap.cc:654] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.982822 2014 tablet_bootstrap.cc:654] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.983552 2016 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: Bootstrap starting.
I20260927 00:38:56.984160 2015 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: No bootstrap required, opened a new log
I20260927 00:38:56.984366 2015 ts_tablet_manager.cc:1397] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260927 00:38:56.984484 2016 tablet_bootstrap.cc:654] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:56.984863 2015 raft_consensus.cc:348] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.984989 2015 raft_consensus.cc:374] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.985023 2015 raft_consensus.cc:729] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4f894b28cd4f81b99dee05d36512a7, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.985148 2015 consensus_queue.cc:260] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.985432 2022 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:56.985569 2022 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:56.986119 2014 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: No bootstrap required, opened a new log
I20260927 00:38:56.986235 2014 ts_tablet_manager.cc:1397] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:38:56.986315 2016 tablet_bootstrap.cc:492] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: No bootstrap required, opened a new log
I20260927 00:38:56.986388 2016 ts_tablet_manager.cc:1397] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:56.986672 2014 raft_consensus.cc:348] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.986788 2014 raft_consensus.cc:374] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.986822 2014 raft_consensus.cc:729] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a77807fccd4a1bb4d3a2d419599d05, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.986820 2016 raft_consensus.cc:348] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.986936 2016 raft_consensus.cc:374] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:56.986969 2016 raft_consensus.cc:729] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d86f4ca5eae4164a009306a75e87c8e, State: Initialized, Role: FOLLOWER
I20260927 00:38:56.986937 2014 consensus_queue.cc:260] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.987087 2016 consensus_queue.cc:260] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:56.987239 2014 ts_tablet_manager.cc:1428] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:38:56.987205 2021 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:56.987352 2016 ts_tablet_manager.cc:1428] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:38:56.987366 2021 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:56.987876 2044 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:56.987998 2044 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:56.988955 2015 ts_tablet_manager.cc:1428] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260927 00:38:57.108764 2022 raft_consensus.cc:482] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:57.108903 2022 raft_consensus.cc:504] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:57.109331 2022 leader_election.cc:288] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615), 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541)
I20260927 00:38:57.109740 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7825cdea94a02996f4095e476b9a4" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d86f4ca5eae4164a009306a75e87c8e" is_pre_election: true
I20260927 00:38:57.109745 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7825cdea94a02996f4095e476b9a4" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05" is_pre_election: true
I20260927 00:38:57.109866 1945 raft_consensus.cc:2454] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 0.
I20260927 00:38:57.109905 1805 raft_consensus.cc:2454] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 0.
I20260927 00:38:57.110154 1597 leader_election.cc:302] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d86f4ca5eae4164a009306a75e87c8e, ac4f894b28cd4f81b99dee05d36512a7; no voters:
I20260927 00:38:57.110325 2022 raft_consensus.cc:2781] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:57.110383 2022 raft_consensus.cc:482] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:57.110407 2022 raft_consensus.cc:3037] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:57.111382 2022 raft_consensus.cc:504] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:57.111714 2022 leader_election.cc:288] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 election: Requested vote from peers 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615), 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541)
I20260927 00:38:57.112046 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7825cdea94a02996f4095e476b9a4" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a77807fccd4a1bb4d3a2d419599d05"
I20260927 00:38:57.112063 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45e7825cdea94a02996f4095e476b9a4" candidate_uuid: "ac4f894b28cd4f81b99dee05d36512a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d86f4ca5eae4164a009306a75e87c8e"
I20260927 00:38:57.112178 1945 raft_consensus.cc:3037] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:57.112154 1805 raft_consensus.cc:3037] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:57.113207 1805 raft_consensus.cc:2454] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 1.
I20260927 00:38:57.113529 1598 leader_election.cc:302] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a77807fccd4a1bb4d3a2d419599d05, ac4f894b28cd4f81b99dee05d36512a7; no voters:
I20260927 00:38:57.113692 2022 raft_consensus.cc:2781] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:57.113698 1945 raft_consensus.cc:2454] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac4f894b28cd4f81b99dee05d36512a7 in term 1.
I20260927 00:38:57.113790 2022 raft_consensus.cc:686] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 LEADER]: Becoming Leader. State: Replica: ac4f894b28cd4f81b99dee05d36512a7, State: Running, Role: LEADER
I20260927 00:38:57.113938 2022 consensus_queue.cc:237] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } }
I20260927 00:38:57.114629 2025 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: TxnStatusTablet state changed. Reason: New leader ac4f894b28cd4f81b99dee05d36512a7. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.114724 2025 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:57.114974 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:38:57.115111 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:38:57.115826 1502 catalog_manager.cc:5495] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } health_report { overall_health: UNKNOWN } } }
I20260927 00:38:57.148845 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.145630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.0.98.1:46001)
0927 00:38:57.145744 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.145748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.145762 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:38:57.145936 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.145939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.146001 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.146066 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:57.146292 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.146869 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.147786 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.148009 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.148095 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.148418 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.148490 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.148548 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.148649 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.148701 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:38:57.148845 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.145533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46001 (local address 127.0.0.1:51772)
0927 00:38:57.145681 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:57.145697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:57.145813 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:57.146092 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:57.146095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:57.146104 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:38:57.146198 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.146203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.146968 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.146972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.147680 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.147689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.148111 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.148114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.148228 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.148283 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.148565 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.148580 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 00:38:57.148639 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 00:38:57.148724 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 00:38:57.151705 1805 raft_consensus.cc:1261] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Refusing update from remote peer ac4f894b28cd4f81b99dee05d36512a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:57.151793 1945 raft_consensus.cc:1261] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Refusing update from remote peer ac4f894b28cd4f81b99dee05d36512a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:57.152187 2025 consensus_queue.cc:1035] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:57.152429 2022 consensus_queue.cc:1035] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:57.154312 2044 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: TxnStatusTablet state changed. Reason: New leader ac4f894b28cd4f81b99dee05d36512a7. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.154465 2044 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:57.154984 2021 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: TxnStatusTablet state changed. Reason: New leader ac4f894b28cd4f81b99dee05d36512a7. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.155109 2021 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:57.156013 2025 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.156154 2025 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:57.157653 2022 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.157797 2022 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P ac4f894b28cd4f81b99dee05d36512a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:57.158583 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333763224169111552 to 7333763224019951616. Current Snapshot: MvccSnapshot[applied={T|T < 7333763224169111552}]
I20260927 00:38:57.159591 2021 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.159708 2021 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 83a77807fccd4a1bb4d3a2d419599d05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:57.161311 2044 tablet_replica.cc:439] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac4f894b28cd4f81b99dee05d36512a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 } } peers { permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 } } peers { permanent_uuid: "9d86f4ca5eae4164a009306a75e87c8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } }
I20260927 00:38:57.161459 2044 tablet_replica.cc:442] T 45e7825cdea94a02996f4095e476b9a4 P 9d86f4ca5eae4164a009306a75e87c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:57.167373 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.163801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.0.98.3:36541)
0927 00:38:57.163909 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.163914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.164046 (+ 132us) server_negotiation.cc:407] Connection header received
0927 00:38:57.164193 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.164220 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.164283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.164373 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:57.164585 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.165082 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.165928 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.166180 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.166266 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.166515 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.166997 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.167093 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.167186 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.167240 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:38:57.178609 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.175354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46695 (local address 127.0.98.1:46001)
0927 00:38:57.175482 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.175486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.175501 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:57.175623 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.175626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.175670 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.175730 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:57.175978 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.176616 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.177599 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.177821 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.177910 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.178180 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.178256 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.178313 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.178406 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.178469 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:38:57.178609 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.175229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46001 (local address 127.0.98.3:46695)
0927 00:38:57.175420 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:57.175435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:57.175524 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:57.175752 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:57.175754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:57.175760 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:38:57.175884 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.175889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.176719 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.176723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.177473 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.177483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.177917 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.177920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.178010 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.178055 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.178324 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.178330 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:38:57.178393 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:38:57.178469 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 00:38:57.186178 1805 raft_consensus.cc:1261] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Refusing update from remote peer 9d86f4ca5eae4164a009306a75e87c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:57.186324 1665 raft_consensus.cc:1261] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Refusing update from remote peer 9d86f4ca5eae4164a009306a75e87c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:57.186784 2044 consensus_queue.cc:1035] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a77807fccd4a1bb4d3a2d419599d05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:57.186981 2020 consensus_queue.cc:1035] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac4f894b28cd4f81b99dee05d36512a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:57.212981 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.208465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53069 (local address 127.0.98.3:36541)
0927 00:38:57.208600 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.208604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.208619 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:38:57.208664 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.208667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.208718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.208792 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:57.209054 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.209842 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.211659 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.211971 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.212046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.212507 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.212596 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.212634 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.212765 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.212823 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:38:57.213402 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.207494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36541 (local address 127.0.98.1:53069)
0927 00:38:57.207834 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:57.207851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:57.207960 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:57.208828 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:57.208831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:57.208841 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:38:57.208953 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.208958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.210521 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.210525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.211520 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:57.211531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.212156 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.212159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:57.212303 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.212360 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:38:57.212683 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:38:57.212689 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:38:57.213136 (+ 447us) client_negotiation.cc:240] Negotiation successful
0927 00:38:57.213233 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20260927 00:38:57.245189 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:57.245404 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:57.247895 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab85ea060f974bbf919b9bd6deefa7c4: Sending DeleteTablet for 3 replicas of tablet faf21c27f8704600b0d606dab1c44849
I20260927 00:38:57.248629 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet faf21c27f8704600b0d606dab1c44849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34534
I20260927 00:38:57.248631 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet faf21c27f8704600b0d606dab1c44849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51748
I20260927 00:38:57.248819 2038 tablet_replica.cc:330] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: stopping tablet replica
I20260927 00:38:57.248935 2038 raft_consensus.cc:2229] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:38:57.249065 2038 raft_consensus.cc:2258] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:57.249235 2037 tablet_replica.cc:330] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: stopping tablet replica
I20260927 00:38:57.249353 2037 raft_consensus.cc:2229] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:38:57.249462 2037 raft_consensus.cc:2258] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:57.250326 2038 ts_tablet_manager.cc:1905] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:57.250447 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet faf21c27f8704600b0d606dab1c44849 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45904
I20260927 00:38:57.250633 2039 tablet_replica.cc:330] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: stopping tablet replica
I20260927 00:38:57.250736 2039 raft_consensus.cc:2229] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:38:57.251050 2039 raft_consensus.cc:2258] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:57.252962 2037 ts_tablet_manager.cc:1905] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:57.253434 2038 ts_tablet_manager.cc:1918] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:57.253506 2038 log.cc:1196] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/faf21c27f8704600b0d606dab1c44849
I20260927 00:38:57.253758 2038 ts_tablet_manager.cc:1939] T faf21c27f8704600b0d606dab1c44849 P 83a77807fccd4a1bb4d3a2d419599d05: Deleting consensus metadata
I20260927 00:38:57.255018 2039 ts_tablet_manager.cc:1905] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:57.255730 2037 ts_tablet_manager.cc:1918] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:57.255796 2037 log.cc:1196] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/faf21c27f8704600b0d606dab1c44849
I20260927 00:38:57.256011 2037 ts_tablet_manager.cc:1939] T faf21c27f8704600b0d606dab1c44849 P ac4f894b28cd4f81b99dee05d36512a7: Deleting consensus metadata
I20260927 00:38:57.257035 1493 catalog_manager.cc:4841] TS ac4f894b28cd4f81b99dee05d36512a7 (127.0.98.1:46001): tablet faf21c27f8704600b0d606dab1c44849 (table test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]) successfully deleted
I20260927 00:38:57.258131 2039 ts_tablet_manager.cc:1918] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:57.258203 2039 log.cc:1196] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/faf21c27f8704600b0d606dab1c44849
I20260927 00:38:57.258437 2039 ts_tablet_manager.cc:1939] T faf21c27f8704600b0d606dab1c44849 P 9d86f4ca5eae4164a009306a75e87c8e: Deleting consensus metadata
I20260927 00:38:57.259446 1494 catalog_manager.cc:4841] TS 83a77807fccd4a1bb4d3a2d419599d05 (127.0.98.2:35615): tablet faf21c27f8704600b0d606dab1c44849 (table test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]) successfully deleted
I20260927 00:38:57.260371 1493 catalog_manager.cc:4841] TS 9d86f4ca5eae4164a009306a75e87c8e (127.0.98.3:36541): tablet faf21c27f8704600b0d606dab1c44849 (table test-table [id=17ad7a7e4a5d40a69e552a05244cc22b]) successfully deleted
Sep 27 00:38:57 dist-test-slave-xltq krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469536, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:38:57.270562 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.257576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.98.1:46001)
0927 00:38:57.257687 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.257691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.258268 (+ 577us) server_negotiation.cc:407] Connection header received
0927 00:38:57.258690 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.258693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.258742 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.258791 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:57.259827 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.260624 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.261832 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.262182 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.262362 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.266229 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:57.266247 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:57.266267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:57.266310 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:57.268398 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:57.268995 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:57.269003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:57.269005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:57.269106 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:57.269414 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:57.269417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:57.269419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:57.269759 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:57.270014 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.270196 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.270363 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:38:57.271044 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51782
I20260927 00:38:57.282608 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:57.273000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.0.98.62:46331)
0927 00:38:57.273164 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:57.273168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:57.273641 (+ 473us) server_negotiation.cc:407] Connection header received
0927 00:38:57.273743 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:57.273746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:57.273801 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:57.273888 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:57.274703 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.275574 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.276797 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:57.277159 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:57.277271 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:57.278856 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:57.278860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:57.278863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:57.278894 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:57.280609 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:57.281133 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:57.281136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:57.281138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:57.281295 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:57.281572 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:57.281575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:57.281577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:57.281859 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:57.282011 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:57.282289 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:38:57.282420 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 00:38:57.283018 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49016
I20260927 00:38:57.284467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1580
I20260927 00:38:57.292725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1720
I20260927 00:38:57.298197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1860
I20260927 00:38:57.303445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 1476
2026-09-27T00:38:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2694 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:38:57 dist-test-slave-xltq krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:38:57 dist-test-slave-xltq krb5kdc[2077](info): set up 2 sockets
Sep 27 00:38:57 dist-test-slave-xltq krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260927 00:38:58.263902 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:38:58 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:38:58.392095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:58.502782 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:58.503088 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:58.503145 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:58.507185 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:38:58.507257 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:38:58.507280 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:58.507303 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:58.507323 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:38:58.507344 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:38:58.512871 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44131
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:58.514009 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:58.515051 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:58.523900 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:58.523918 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:58.523943 2090 server_base.cc:1034] running on GCE node
W20260927 00:38:58.523893 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:58.524870 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:58.526141 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:58.527324 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790469538527302 us; error 38 us; skew 500 ppm
Sep 27 00:38:58 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:58.534118 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:38:58.535135 2090 webserver.cc:460] Webserver started at http://127.0.98.62:35535/ using document root <none> and password file <none>
I20260927 00:38:58.535722 2090 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:58.535791 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:58.535983 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:58.538437 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "3b87175b2b9445ed890810cc0bc0b09d"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "ce2e5483bb91c14eed29c0ef233f6e71"
server_key_iv: "e35617b3b3d2eb66682d779050442b08"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.539028 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b87175b2b9445ed890810cc0bc0b09d"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "ce2e5483bb91c14eed29c0ef233f6e71"
server_key_iv: "e35617b3b3d2eb66682d779050442b08"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.543267 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:38:58.546115 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:58.547585 2090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:58.547711 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "3b87175b2b9445ed890810cc0bc0b09d"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "ce2e5483bb91c14eed29c0ef233f6e71"
server_key_iv: "e35617b3b3d2eb66682d779050442b08"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.547847 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:58.559307 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:58.563248 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:58.563455 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:58.571405 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44131
I20260927 00:38:58.571478 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44131 every 8 connection(s)
I20260927 00:38:58.572474 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:38:58.575390 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:58.578888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2090
I20260927 00:38:58.579026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:38:58.579288 392 external_mini_cluster.cc:1428] Setting key e4047ea991bbeb64c703eac50915445b
I20260927 00:38:58.582262 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Bootstrap starting.
I20260927 00:38:58.585259 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:58.586072 2168 log.cc:824] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:58.588369 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: No bootstrap required, opened a new log
Sep 27 00:38:58 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:58.592754 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } }
I20260927 00:38:58.593115 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:58.593179 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b87175b2b9445ed890810cc0bc0b09d, State: Initialized, Role: FOLLOWER
I20260927 00:38:58.593709 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } }
I20260927 00:38:58.593879 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:38:58.593945 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:38:58.594035 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:58.595252 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } }
I20260927 00:38:58.595328 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:58.580818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.98.62:44131)
0927 00:38:58.581319 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:58.581340 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:58.581384 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:38:58.582240 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:58.582266 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:58.582784 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:58.583195 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:58.583926 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.585431 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.586302 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.586895 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.587054 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:58.589741 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:58.589748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:58.589766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:58.589799 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:58.591830 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.592358 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.592365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.592372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.592528 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.592862 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.592865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.592867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.593423 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:58.593631 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:58.593941 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 00:38:58.594275 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":138,"threads_started":1}
I20260927 00:38:58.595642 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b87175b2b9445ed890810cc0bc0b09d; no voters:
I20260927 00:38:58.595991 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:38:58.596179 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:58.596474 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [term 1 LEADER]: Becoming Leader. State: Replica: 3b87175b2b9445ed890810cc0bc0b09d, State: Running, Role: LEADER
I20260927 00:38:58.596822 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } }
I20260927 00:38:58.598384 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:38:58.598526 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b87175b2b9445ed890810cc0bc0b09d. Latest consensus state: current_term: 1 leader_uuid: "3b87175b2b9445ed890810cc0bc0b09d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } } }
I20260927 00:38:58.598690 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:58.598553 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b87175b2b9445ed890810cc0bc0b09d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b87175b2b9445ed890810cc0bc0b09d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44131 } } }
I20260927 00:38:58.598771 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d [sys.catalog]: This master's current role is: LEADER
I20260927 00:38:58.599071 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:38:58.601879 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:38:58.607837 2177 catalog_manager.cc:1347] Generated new cluster ID: 9a9052099ae94ee49a9cc4ee7b719c33
I20260927 00:38:58.607932 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:38:58.620184 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:38:58.621317 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:38:58.626948 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Generated new TSK 0
I20260927 00:38:58.627868 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:38:58.657852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44131
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:58.777493 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:58.777971 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:58.778108 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:58.784286 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:58.784370 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:58.784431 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:38:58.784459 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:58.784514 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:38:58.792321 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:58.793807 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:58.795164 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:58.805280 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:58.805352 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:58.805408 2194 server_base.cc:1034] running on GCE node
W20260927 00:38:58.805642 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:58.806034 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:58.806715 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:58.807897 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790469538807863 us; error 48 us; skew 500 ppm
Sep 27 00:38:58 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:58.810829 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:38:58.811820 2194 webserver.cc:460] Webserver started at http://127.0.98.1:42327/ using document root <none> and password file <none>
I20260927 00:38:58.812433 2194 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:58.812505 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:58.812716 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:58.814872 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6302ac49f5640ccaef46d26a3978ddd"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "361e961ff81f056674a6a934147150e4"
server_key_iv: "e5b1c2cca7d5bf48fa4448083dd459d3"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.815457 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6302ac49f5640ccaef46d26a3978ddd"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "361e961ff81f056674a6a934147150e4"
server_key_iv: "e5b1c2cca7d5bf48fa4448083dd459d3"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.819687 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:38:58.822520 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:58.823925 2194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:58.824043 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "d6302ac49f5640ccaef46d26a3978ddd"
format_stamp: "Formatted at 2026-09-27 00:38:58 on dist-test-slave-xltq"
server_key: "361e961ff81f056674a6a934147150e4"
server_key_iv: "e5b1c2cca7d5bf48fa4448083dd459d3"
server_key_version: "encryptionkey@0"
I20260927 00:38:58.824162 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:58.850459 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:58.853742 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:58.853924 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:58.855445 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:58.855497 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:58.855552 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:58.855581 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:38:58 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:58.865968 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42865
I20260927 00:38:58.866063 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42865 every 8 connection(s)
I20260927 00:38:58.867241 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:38:58.869930 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:58.856368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60479 (local address 127.0.98.62:44131)
0927 00:38:58.856485 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:58.856489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:58.857607 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 00:38:58.858870 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:58.858874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:58.858937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:58.859013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:58.860411 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.860994 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.862674 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.863015 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.863124 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:58.866418 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:58.866422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:58.866425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:58.866453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:58.867913 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.868546 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.868550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.868552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.868641 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.868932 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.868935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.868936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.869190 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:58.869269 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:58.869704 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 00:38:58.869810 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:38:58.870710 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:58.856945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.1:60479)
0927 00:38:58.857436 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:58.857486 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:58.858616 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:58.859273 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:58.859281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:58.859688 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:58.860238 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:58.860259 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.861127 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.861131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:58.862513 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:58.862526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.863208 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.863212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:58.863436 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:58.864300 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:58.864323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:58.866229 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:58.868084 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:58.868090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:58.868105 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:58.868426 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:58.868720 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:58.868722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:58.868723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:58.868848 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:58.869268 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:58.869273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:58.869498 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:38:58.869665 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 00:38:58.869927 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260927 00:38:58.871915 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44131
I20260927 00:38:58.872092 2325 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:58.872687 2325 heartbeater.cc:511] Master 127.0.98.62:44131 requested a full tablet report, sending...
I20260927 00:38:58.874351 2119 ts_manager.cc:194] Registered new tserver with Master: d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865)
I20260927 00:38:58.874639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2194
I20260927 00:38:58.874728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:38:58.875039 392 external_mini_cluster.cc:1428] Setting key 1c34bc35d2352f4c5e8c831e3e5b7ace
I20260927 00:38:58.875666 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60479
I20260927 00:38:58.882413 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:58.874308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33759 (local address 127.0.98.62:44131)
0927 00:38:58.874445 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:58.874449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:58.875567 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 00:38:58.875704 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:58.875707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:58.875755 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:58.875826 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:58.876434 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.876995 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.878043 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.878269 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.878353 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:58.879451 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:58.879455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:58.879458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:58.879483 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:58.880890 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.881289 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.881293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.881295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.881369 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:58.881611 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:58.881622 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:58.881624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:58.881805 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:58.881888 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:58.882136 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:38:58.882241 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:38:58.882432 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:58.874228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.1:33759)
0927 00:38:58.875467 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:58.875482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:58.875599 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:58.875931 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:58.875934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:58.876156 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:58.876323 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:58.876331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.877132 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.877135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:58.877929 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:58.877940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:58.878367 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:58.878370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:58.878456 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:58.878960 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:58.878974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:58.879312 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:58.880996 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:58.880999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:58.881001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:58.881205 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:58.881447 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:58.881450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:58.881452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:58.881534 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:58.881891 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:58.881894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:58.881989 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 00:38:58.882075 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:38:58.882241 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:38:58.903656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44131
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:59.016328 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:59.016618 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:59.016666 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:59.020848 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:59.020925 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:59.020968 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:38:59.020988 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:59.021032 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:38:59.026634 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:59.027779 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:59.028887 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:59.039309 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:59.039366 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:59.039366 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:59.039556 2334 server_base.cc:1034] running on GCE node
I20260927 00:38:59.040164 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:59.040907 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:59.042073 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790469539042043 us; error 45 us; skew 500 ppm
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:59.045056 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:38:59.046059 2334 webserver.cc:460] Webserver started at http://127.0.98.2:37607/ using document root <none> and password file <none>
I20260927 00:38:59.046640 2334 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:59.046703 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:59.046883 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:59.049096 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "4c7994587ae49cd955aaab9316a9b706"
server_key_iv: "572b1e6cc4f758c595617cb329bb6430"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.049715 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "4c7994587ae49cd955aaab9316a9b706"
server_key_iv: "572b1e6cc4f758c595617cb329bb6430"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.053817 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:38:59.056559 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:59.058056 2334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:38:59.058188 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "4c7994587ae49cd955aaab9316a9b706"
server_key_iv: "572b1e6cc4f758c595617cb329bb6430"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.058320 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:59.098898 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:59.102399 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:59.102567 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:59.104043 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:59.104103 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:59.104161 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:59.104192 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:59.114537 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36517
I20260927 00:38:59.114612 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36517 every 8 connection(s)
I20260927 00:38:59.115628 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:38:59.118389 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.104975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49443 (local address 127.0.98.62:44131)
0927 00:38:59.105123 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.105128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.106091 (+ 963us) server_negotiation.cc:407] Connection header received
0927 00:38:59.107281 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.107285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.107354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.107462 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.108956 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.109790 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.111199 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.111527 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.111644 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.114384 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.114388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.114391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.114423 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.116091 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.116743 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.116747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.116749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.116865 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.117253 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.117257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.117258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.117474 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.117565 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.118060 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.118264 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:38:59.119171 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.105424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.2:49443)
0927 00:38:59.105909 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.105963 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.107000 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.107686 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.107695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.108191 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:59.108782 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.108803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.109941 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.109945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.111077 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.111088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.111675 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.111678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.111825 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.112536 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:59.112552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:59.114197 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:59.116252 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.116259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.116274 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.116618 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.116950 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.116952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.116954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.117092 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.117584 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:59.117589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:59.117827 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.117980 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.118296 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260927 00:38:59.120750 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44131
I20260927 00:38:59.120819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2334
I20260927 00:38:59.120919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:38:59.120980 2465 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:59.121246 392 external_mini_cluster.cc:1428] Setting key 6653be7250ceb6f37f8081b93c839d2c
I20260927 00:38:59.121747 2465 heartbeater.cc:511] Master 127.0.98.62:44131 requested a full tablet report, sending...
I20260927 00:38:59.122864 2121 ts_manager.cc:194] Registered new tserver with Master: 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:38:59.123612 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49443
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:38:59.131282 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.123289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.2:58117)
0927 00:38:59.123596 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.123610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.123717 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.124012 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.124015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.124283 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:59.124422 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.124429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.125413 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.125417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.126232 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.126244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.126705 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.126709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.126813 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.127504 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:59.127519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:59.127975 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:59.129762 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.129766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.129769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.130016 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.130330 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.130333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.130335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.130428 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.130826 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:59.130830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:59.130919 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.131003 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.131131 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260927 00:38:59.131305 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.123329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58117 (local address 127.0.98.62:44131)
0927 00:38:59.123439 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.123443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.123691 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:38:59.123838 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.123841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.123891 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.123976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.124540 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.125281 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.126351 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.126572 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.126694 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.128171 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.128174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.128177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.128234 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.129647 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.130170 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.130173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.130174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.130248 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.130504 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.130506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.130507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.130737 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.130814 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.131041 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.131188 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:38:59.152287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44131
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:38:59.269014 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:38:59.269322 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:38:59.269371 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:38:59.274070 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:38:59.274150 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:38:59.274199 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:38:59.274212 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:38:59.274250 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:38:59.279783 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:38:59.280993 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:38:59.282131 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:38:59.291903 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:59.291906 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:38:59.291903 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:38:59.292153 2474 server_base.cc:1034] running on GCE node
I20260927 00:38:59.292723 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:38:59.293371 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:38:59.294546 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790469539294508 us; error 50 us; skew 500 ppm
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:38:59.297333 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:38:59.298367 2474 webserver.cc:460] Webserver started at http://127.0.98.3:39507/ using document root <none> and password file <none>
I20260927 00:38:59.298934 2474 fs_manager.cc:359] Metadata directory not provided
I20260927 00:38:59.299012 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:38:59.299209 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:38:59.301425 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "af0d88bfc20e47a8b4e0a88f8fc59312"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "d687ea0c9cd036e097897a879bd39dfd"
server_key_iv: "26da4bdd81c9689003c0c27c6ee68955"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.301959 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "af0d88bfc20e47a8b4e0a88f8fc59312"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "d687ea0c9cd036e097897a879bd39dfd"
server_key_iv: "26da4bdd81c9689003c0c27c6ee68955"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.306034 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:38:59.308761 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:59.310184 2474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:38:59.310331 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "af0d88bfc20e47a8b4e0a88f8fc59312"
format_stamp: "Formatted at 2026-09-27 00:38:59 on dist-test-slave-xltq"
server_key: "d687ea0c9cd036e097897a879bd39dfd"
server_key_iv: "26da4bdd81c9689003c0c27c6ee68955"
server_key_version: "encryptionkey@0"
I20260927 00:38:59.310446 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:38:59.339715 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:38:59.343207 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:38:59.343393 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:38:59.344949 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:38:59.345002 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:59.345057 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:38:59.345085 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:38:59.354470 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36017
I20260927 00:38:59.354506 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36017 every 8 connection(s)
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:59.355553 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:38:59.358965 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.345764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60765 (local address 127.0.98.62:44131)
0927 00:38:59.345879 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.345882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.346994 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 00:38:59.348015 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.348019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.348082 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.348175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.349803 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.350610 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.352302 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.352631 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.352768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.355529 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.355533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.355535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.355559 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.357134 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.357604 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.357607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.357608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.357680 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.357973 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.357975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.357977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.358146 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.358252 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.358716 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.358837 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:38:59.359861 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.346322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.3:60765)
0927 00:38:59.346841 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.346887 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.347792 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.348491 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.348500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.348945 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:59.349630 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.349649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.350753 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.350757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.352137 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.352146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.352809 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.352814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.353006 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.353708 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:59.353724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:59.355359 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:59.357271 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.357276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.357287 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.357526 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.357778 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.357781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.357782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.357908 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.358265 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:59.358270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:59.358512 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.358667 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.358994 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260927 00:38:59.361104 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44131
I20260927 00:38:59.361285 2605 heartbeater.cc:464] Registering TS with master...
I20260927 00:38:59.361842 2605 heartbeater.cc:511] Master 127.0.98.62:44131 requested a full tablet report, sending...
I20260927 00:38:59.362977 2119 ts_manager.cc:194] Registered new tserver with Master: af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017)
I20260927 00:38:59.363161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2474
I20260927 00:38:59.363276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:38:59.363574 392 external_mini_cluster.cc:1428] Setting key fcadc026b6fa1ccabda350adb1f9b7d7
I20260927 00:38:59.363760 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60765
I20260927 00:38:59.365530 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:38:59.372850 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.363573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59513 (local address 127.0.98.62:44131)
0927 00:38:59.364053 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.364057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.364069 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:38:59.364170 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.364173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.364247 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.364309 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.366429 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.367120 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.367939 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.368158 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.368280 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.369544 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.369547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.369549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.369572 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.371191 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.371627 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.371631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.371633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.371728 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.372001 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.372004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.372006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.372233 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.372325 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.372557 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.372679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260927 00:38:59.372867 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.363516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.98.3:59513)
0927 00:38:59.363983 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.363997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.364086 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.364336 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.364339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.366108 (+ 1769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:59.366233 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.366243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.367256 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.367260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.367829 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.367837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.368281 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.368285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.368370 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.368923 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:59.368937 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:59.369385 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:59.371306 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.371309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.371310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.371541 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.371806 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.371808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.371809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.371882 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.372316 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:59.372319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:59.372405 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.372526 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.372681 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":103,"threads_started":1}
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:59.392560 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.384466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.0.98.62:44131)
0927 00:38:59.384599 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.384602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.384616 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:38:59.384655 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.384658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.384696 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.384765 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.385515 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.386155 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.387044 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.387280 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.387374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.389779 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.389782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.389784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.389810 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.391148 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.391502 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.391505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.391506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.391584 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.391776 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.391778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.391779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.391957 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.392054 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.392310 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.392403 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:38:59.395494 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:38:59.398628 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:59.413280 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.407346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.98.2:36517)
0927 00:38:59.408569 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.408580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.408601 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:38:59.408667 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.408672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.408830 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.409109 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.409302 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410514 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.411868 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.412683 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.412793 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.412966 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.413025 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.413119 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":122,"threads_started":1}
I20260927 00:38:59.413339 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.407243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36517 (local address 127.0.0.1:57608)
0927 00:38:59.407856 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.407906 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.408046 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.408985 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.408991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.409015 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.409197 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.409204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410639 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.410643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.411763 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.411772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.412784 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.412793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.412881 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.412913 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.412997 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.413115 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":57,"thread_start_us":85,"threads_started":1}
I20260927 00:38:59.413432 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.408474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.0.98.3:36017)
0927 00:38:59.409100 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.409110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.409132 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:38:59.409213 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.409218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.409394 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.409515 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.409677 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410382 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.412116 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.412927 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.413038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.413186 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.413227 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.413309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":81,"threads_started":1}
I20260927 00:38:59.413821 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.408337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.0.1:58420)
0927 00:38:59.408802 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.408813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.408889 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.409503 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.409506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.409523 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.409596 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.409601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410483 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.410486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.412006 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.412013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.413035 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.413038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.413112 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.413131 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.413187 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.413245 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":114,"thread_start_us":86,"threads_started":1}
I20260927 00:38:59.415035 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.407398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42865 (local address 127.0.0.1:59614)
0927 00:38:59.408176 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.408189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.408307 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.408984 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.408991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.409023 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.409169 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.409176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410604 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.410607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.411997 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.412007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.414659 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.414663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.414747 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.414769 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.414835 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.414902 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":121,"thread_start_us":81,"threads_started":1}
I20260927 00:38:59.415417 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.407587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.0.98.1:42865)
0927 00:38:59.408050 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.408057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.408290 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:38:59.408443 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.408450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.408677 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.408821 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.409298 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.410489 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.413564 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.414346 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.414447 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.414590 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.415204 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.415302 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":85,"threads_started":1}
I20260927 00:38:59.416684 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7066dc5d504c4ae99119d28783f50684 (DEFAULT_TABLE table=test-table [id=0551a3160d6b4f78801fec3ecbf69912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.416603 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 7066dc5d504c4ae99119d28783f50684 (DEFAULT_TABLE table=test-table [id=0551a3160d6b4f78801fec3ecbf69912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.417611 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7066dc5d504c4ae99119d28783f50684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.417677 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7066dc5d504c4ae99119d28783f50684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.418179 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 7066dc5d504c4ae99119d28783f50684 (DEFAULT_TABLE table=test-table [id=0551a3160d6b4f78801fec3ecbf69912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.419196 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7066dc5d504c4ae99119d28783f50684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.424605 2628 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Bootstrap starting.
I20260927 00:38:59.426316 2629 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Bootstrap starting.
I20260927 00:38:59.427316 2630 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Bootstrap starting.
I20260927 00:38:59.427379 2628 tablet_bootstrap.cc:654] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.428313 2628 log.cc:824] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:59.428570 2629 tablet_bootstrap.cc:654] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.429322 2630 tablet_bootstrap.cc:654] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.429301 2629 log.cc:824] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:59.430096 2630 log.cc:824] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:38:59.430385 2628 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: No bootstrap required, opened a new log
I20260927 00:38:59.430576 2628 ts_tablet_manager.cc:1397] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:38:59.431226 2629 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: No bootstrap required, opened a new log
I20260927 00:38:59.431407 2629 ts_tablet_manager.cc:1397] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:38:59.431677 2630 tablet_bootstrap.cc:492] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: No bootstrap required, opened a new log
I20260927 00:38:59.432099 2630 ts_tablet_manager.cc:1397] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:38:59.435045 2629 raft_consensus.cc:348] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.435045 2628 raft_consensus.cc:348] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.435333 2628 raft_consensus.cc:374] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.435333 2629 raft_consensus.cc:374] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.435382 2628 raft_consensus.cc:729] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.435391 2629 raft_consensus.cc:729] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.435356 2630 raft_consensus.cc:348] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.435612 2630 raft_consensus.cc:374] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.435667 2630 raft_consensus.cc:729] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.435883 2628 consensus_queue.cc:260] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.435883 2629 consensus_queue.cc:260] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.436069 2630 consensus_queue.cc:260] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.436726 2325 heartbeater.cc:503] Master 127.0.98.62:44131 was elected leader, sending a full tablet report...
I20260927 00:38:59.436954 2629 ts_tablet_manager.cc:1428] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:38:59.436954 2628 ts_tablet_manager.cc:1428] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:38:59.437301 2605 heartbeater.cc:503] Master 127.0.98.62:44131 was elected leader, sending a full tablet report...
I20260927 00:38:59.438088 2630 ts_tablet_manager.cc:1428] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:38:59.438253 2465 heartbeater.cc:503] Master 127.0.98.62:44131 was elected leader, sending a full tablet report...
W20260927 00:38:59.606652 2606 tablet.cc:2369] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:59.612504 2634 raft_consensus.cc:482] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:59.612761 2634 raft_consensus.cc:504] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.613967 2634 leader_election.cc:288] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517), d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865)
W20260927 00:38:59.616948 2466 tablet.cc:2369] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:59.618458 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.614417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44733 (local address 127.0.98.1:42865)
0927 00:38:59.614540 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.614544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.614638 (+ 94us) server_negotiation.cc:407] Connection header received
0927 00:38:59.614824 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.614828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.614877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.614946 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.615308 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.615994 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.617392 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.617878 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.617979 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.618046 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.618250 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.618331 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:38:59.618458 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.614318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42865 (local address 127.0.98.3:44733)
0927 00:38:59.614540 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.614556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.614676 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.614960 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.614962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.614979 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.615111 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.615119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.616092 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.616095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.617274 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.617285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.617999 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.618002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.618105 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.618124 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.618221 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.618304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":24}
W20260927 00:38:59.618835 2326 tablet.cc:2369] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:38:59.619201 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7066dc5d504c4ae99119d28783f50684" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd" is_pre_election: true
I20260927 00:38:59.619302 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.614417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47503 (local address 127.0.98.2:36517)
0927 00:38:59.614540 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.614544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.614844 (+ 300us) server_negotiation.cc:407] Connection header received
0927 00:38:59.614953 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.614956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.615007 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.615089 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.615864 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.616606 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.618128 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.618565 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.618684 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.618753 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.619091 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.619176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:38:59.619302 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.614318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36517 (local address 127.0.98.3:47503)
0927 00:38:59.614744 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.614757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.614859 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.615142 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.615145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.615161 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.615704 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.615716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.616785 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.616789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.617968 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.617980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.618833 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.618837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.618944 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.618971 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.619056 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.619132 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":82,"thread_start_us":168,"threads_started":1}
I20260927 00:38:59.619551 2270 raft_consensus.cc:2454] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.620012 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7066dc5d504c4ae99119d28783f50684" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" is_pre_election: true
I20260927 00:38:59.620064 2493 leader_election.cc:302] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:38:59.620354 2419 raft_consensus.cc:2454] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.620386 2634 raft_consensus.cc:2781] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:59.620479 2634 raft_consensus.cc:482] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:59.620525 2634 raft_consensus.cc:3037] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.621739 2634 raft_consensus.cc:504] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.622114 2634 leader_election.cc:288] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Requested vote from peers 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517), d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865)
I20260927 00:38:59.622476 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7066dc5d504c4ae99119d28783f50684" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
I20260927 00:38:59.622614 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7066dc5d504c4ae99119d28783f50684" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd"
I20260927 00:38:59.622613 2419 raft_consensus.cc:3037] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.622745 2270 raft_consensus.cc:3037] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.623910 2419 raft_consensus.cc:2454] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:38:59.623946 2270 raft_consensus.cc:2454] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:38:59.624284 2492 leader_election.cc:302] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, af0d88bfc20e47a8b4e0a88f8fc59312; no voters:
I20260927 00:38:59.624496 2634 raft_consensus.cc:2781] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:59.624681 2634 raft_consensus.cc:686] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 LEADER]: Becoming Leader. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Running, Role: LEADER
I20260927 00:38:59.625032 2634 consensus_queue.cc:237] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.628460 2122 catalog_manager.cc:5495] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 reported cstate change: term changed from 0 to 1, leader changed from <none> to af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: HEALTHY } } }
Sep 27 00:38:59 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:38:59.678200 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.670045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.98.62:44131)
0927 00:38:59.670201 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.670206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.670406 (+ 200us) server_negotiation.cc:407] Connection header received
0927 00:38:59.670523 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.670526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.670564 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.670627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:38:59.671301 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.671888 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.672666 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.672891 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.672981 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.675211 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:38:59.675215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:38:59.675218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:38:59.675240 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:38:59.676578 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.677018 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.677021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.677023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.677093 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:38:59.677327 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:38:59.677330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:38:59.677332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:38:59.677514 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:38:59.677595 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.677886 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.678071 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":72}
I20260927 00:38:59.678201 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.669993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44131 (local address 127.0.0.1:33520)
0927 00:38:59.670318 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.670330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.670441 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.670653 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.670656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.671084 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:38:59.671176 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.671183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.671991 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.671994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.672560 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.672568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.672985 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.672989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.673060 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.673587 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:38:59.673598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:38:59.675062 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:38:59.676678 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.676692 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.676700 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.676931 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.677167 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:38:59.677170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:38:59.677172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:38:59.677254 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:38:59.677608 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:38:59.677617 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:38:59.677730 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.677831 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.677999 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260927 00:38:59.682658 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33520:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:38:59.683492 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:59.688463 2399 tablet_service.cc:1467] Processing CreateTablet for tablet fefa27349afa48cda3b225804685a1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81983806b0e84aad8cb629e8133a2c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:59.688436 2259 tablet_service.cc:1467] Processing CreateTablet for tablet fefa27349afa48cda3b225804685a1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81983806b0e84aad8cb629e8133a2c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:59.688472 2539 tablet_service.cc:1467] Processing CreateTablet for tablet fefa27349afa48cda3b225804685a1d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81983806b0e84aad8cb629e8133a2c69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:38:59.689210 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa27349afa48cda3b225804685a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.689213 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa27349afa48cda3b225804685a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.689211 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fefa27349afa48cda3b225804685a1d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.691747 2629 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: Bootstrap starting.
I20260927 00:38:59.692072 2630 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: Bootstrap starting.
I20260927 00:38:59.692461 2628 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: Bootstrap starting.
I20260927 00:38:59.692778 2629 tablet_bootstrap.cc:654] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.693070 2630 tablet_bootstrap.cc:654] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.693404 2628 tablet_bootstrap.cc:654] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.694419 2630 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: No bootstrap required, opened a new log
I20260927 00:38:59.694497 2630 ts_tablet_manager.cc:1397] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:59.694618 2629 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: No bootstrap required, opened a new log
I20260927 00:38:59.694712 2629 ts_tablet_manager.cc:1397] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:59.694962 2630 raft_consensus.cc:348] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.695079 2630 raft_consensus.cc:374] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.695101 2630 raft_consensus.cc:729] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.695132 2629 raft_consensus.cc:348] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.695262 2629 raft_consensus.cc:374] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.695232 2630 consensus_queue.cc:260] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.695302 2629 raft_consensus.cc:729] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.695429 2629 consensus_queue.cc:260] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.695488 2630 ts_tablet_manager.cc:1428] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:38:59.695498 2636 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.695684 2636 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.695685 2629 ts_tablet_manager.cc:1428] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:38:59.696341 2628 tablet_bootstrap.cc:492] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: No bootstrap required, opened a new log
I20260927 00:38:59.696434 2628 ts_tablet_manager.cc:1397] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:38:59.696393 2635 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.696552 2635 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.696885 2628 raft_consensus.cc:348] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.697002 2628 raft_consensus.cc:374] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.697036 2628 raft_consensus.cc:729] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.697150 2628 consensus_queue.cc:260] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.697397 2628 ts_tablet_manager.cc:1428] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:38:59.697378 2634 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.697496 2634 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.700464 2634 raft_consensus.cc:482] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:59.700636 2634 raft_consensus.cc:504] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.701025 2634 leader_election.cc:288] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:38:59.701367 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa27349afa48cda3b225804685a1d9" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd" is_pre_election: true
I20260927 00:38:59.701352 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa27349afa48cda3b225804685a1d9" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" is_pre_election: true
I20260927 00:38:59.701500 2419 raft_consensus.cc:2454] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.701508 2270 raft_consensus.cc:2454] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.701752 2493 leader_election.cc:302] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:38:59.701920 2634 raft_consensus.cc:2781] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:59.701977 2634 raft_consensus.cc:482] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:59.702003 2634 raft_consensus.cc:3037] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.702955 2634 raft_consensus.cc:504] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.703267 2634 leader_election.cc:288] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Requested vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:38:59.703603 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa27349afa48cda3b225804685a1d9" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd"
I20260927 00:38:59.703625 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fefa27349afa48cda3b225804685a1d9" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
I20260927 00:38:59.703730 2270 raft_consensus.cc:3037] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.703730 2419 raft_consensus.cc:3037] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.704895 2270 raft_consensus.cc:2454] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:38:59.704895 2419 raft_consensus.cc:2454] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:38:59.705170 2492 leader_election.cc:302] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:38:59.705340 2634 raft_consensus.cc:2781] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:59.705423 2634 raft_consensus.cc:686] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 LEADER]: Becoming Leader. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Running, Role: LEADER
I20260927 00:38:59.705554 2634 consensus_queue.cc:237] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.706231 2638 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: TxnStatusTablet state changed. Reason: New leader af0d88bfc20e47a8b4e0a88f8fc59312. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.706350 2638 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:59.706637 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:38:59.706861 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:38:59.707193 2118 catalog_manager.cc:5495] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 reported cstate change: term changed from 0 to 1, leader changed from <none> to af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: HEALTHY } } }
I20260927 00:38:59.724728 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.720688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.0.1:58436)
0927 00:38:59.720852 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.720870 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.720995 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.721419 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.721423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.721431 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:38:59.721536 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.721541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.722399 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.722402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.723032 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.723041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.723487 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.723491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.723575 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.723627 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:38:59.724391 (+ 764us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:38:59.724412 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.724483 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.724583 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:38:59.724728 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.720761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.0.98.3:36017)
0927 00:38:59.720896 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.720901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.720924 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:38:59.721173 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.721177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.721257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.721372 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:59.721673 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.722280 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.723152 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.723372 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.723468 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.723773 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:59.724304 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:59.724435 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.724500 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.724570 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:38:59.730760 2270 raft_consensus.cc:1261] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:59.730752 2419 raft_consensus.cc:1261] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:59.731758 2638 consensus_queue.cc:1035] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:59.732012 2634 consensus_queue.cc:1035] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:59.736303 2636 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: TxnStatusTablet state changed. Reason: New leader af0d88bfc20e47a8b4e0a88f8fc59312. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.736492 2636 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.737200 2635 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: TxnStatusTablet state changed. Reason: New leader af0d88bfc20e47a8b4e0a88f8fc59312. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.737329 2635 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.741699 2638 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.741844 2638 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:59.741969 2638 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.742033 2638 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P af0d88bfc20e47a8b4e0a88f8fc59312: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:38:59.742106 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333763234727849984 to 7333763234634952704. Current Snapshot: MvccSnapshot[applied={T|T < 7333763234727849984}]
I20260927 00:38:59.742576 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333763234727849984 to 7333763234634952704. Current Snapshot: MvccSnapshot[applied={T|T < 7333763234727849984}]
I20260927 00:38:59.742790 2636 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.742893 2636 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P d6302ac49f5640ccaef46d26a3978ddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.742841 2635 tablet_replica.cc:439] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } } }
I20260927 00:38:59.742967 2635 tablet_replica.cc:442] T fefa27349afa48cda3b225804685a1d9 P 9060aa9e4e6649ed9c6b8f203da6a082: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:38:59.746834 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333763234727849984 to 7333763234634952704. Current Snapshot: MvccSnapshot[applied={T|T < 7333763234727849984}]
I20260927 00:38:59.754047 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.750010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.0.98.3:36017)
0927 00:38:59.750117 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.750121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.750157 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:38:59.750378 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.750381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.750434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.750506 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:59.750778 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.751589 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.752777 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.753122 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.753231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.753539 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:59.753633 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:59.753707 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.753836 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.753893 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:38:59.766310 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.763014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.3:60847)
0927 00:38:59.763200 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.763213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.763331 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.763534 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.763536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.763542 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:38:59.763685 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.763691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.764441 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.764445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.765091 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.765101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.765531 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.765534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.765635 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.765686 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:38:59.765990 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:38:59.765996 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.766055 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.766130 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":21}
I20260927 00:38:59.766310 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.763057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60847 (local address 127.0.98.3:36017)
0927 00:38:59.763166 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.763171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.763283 (+ 112us) server_negotiation.cc:407] Connection header received
0927 00:38:59.763422 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.763425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.763464 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.763522 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:38:59.763774 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.764334 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.765210 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.765422 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.765516 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.765849 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:38:59.765924 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:38:59.765981 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.766062 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.766114 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":29}
I20260927 00:38:59.775241 2419 raft_consensus.cc:1261] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:59.775314 2270 raft_consensus.cc:1261] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:38:59.775887 2634 consensus_queue.cc:1035] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:59.776139 2658 consensus_queue.cc:1035] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:38:59.824851 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:59.825032 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:38:59.827801 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Sending DeleteTablet for 3 replicas of tablet 7066dc5d504c4ae99119d28783f50684
I20260927 00:38:59.828732 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7066dc5d504c4ae99119d28783f50684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57608
I20260927 00:38:59.828704 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 7066dc5d504c4ae99119d28783f50684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58420
I20260927 00:38:59.829444 2669 tablet_replica.cc:330] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: stopping tablet replica
I20260927 00:38:59.829492 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:38:59.829576 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 7066dc5d504c4ae99119d28783f50684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:38:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260927 00:38:59.829813 2669 raft_consensus.cc:2229] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 00:38:59.829934 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:38:59.830049 2669 raft_consensus.cc:2258] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:59.830459 2671 tablet_replica.cc:330] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: stopping tablet replica
I20260927 00:38:59.830819 2671 raft_consensus.cc:2229] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:38:59.831072 2671 raft_consensus.cc:2258] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:59.831619 2669 ts_tablet_manager.cc:1905] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:59.833242 2671 ts_tablet_manager.cc:1905] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:59.834754 2669 ts_tablet_manager.cc:1918] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:59.834861 2669 log.cc:1196] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/7066dc5d504c4ae99119d28783f50684
I20260927 00:38:59.835158 2669 ts_tablet_manager.cc:1939] T 7066dc5d504c4ae99119d28783f50684 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting consensus metadata
I20260927 00:38:59.835330 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 83419050085a486fbd270619d8aa2ae3 (DEFAULT_TABLE table=test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.835585 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83419050085a486fbd270619d8aa2ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.836148 2670 tablet_replica.cc:330] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: stopping tablet replica
I20260927 00:38:59.836499 2670 raft_consensus.cc:2229] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:38:59.836920 2670 raft_consensus.cc:2258] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:38:59.837318 2671 ts_tablet_manager.cc:1918] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:59.837464 2671 log.cc:1196] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/7066dc5d504c4ae99119d28783f50684
I20260927 00:38:59.837888 2671 ts_tablet_manager.cc:1939] T 7066dc5d504c4ae99119d28783f50684 P d6302ac49f5640ccaef46d26a3978ddd: Deleting consensus metadata
I20260927 00:38:59.838227 2630 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Bootstrap starting.
I20260927 00:38:59.839016 2630 tablet_bootstrap.cc:654] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.839815 2109 catalog_manager.cc:4841] TS d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865): tablet 7066dc5d504c4ae99119d28783f50684 (table test-table [id=0551a3160d6b4f78801fec3ecbf69912]) successfully deleted
I20260927 00:38:59.839950 2108 catalog_manager.cc:4841] TS 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517): tablet 7066dc5d504c4ae99119d28783f50684 (table test-table [id=0551a3160d6b4f78801fec3ecbf69912]) successfully deleted
I20260927 00:38:59.840108 2630 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: No bootstrap required, opened a new log
I20260927 00:38:59.840179 2630 ts_tablet_manager.cc:1397] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:38:59.840651 2630 raft_consensus.cc:348] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.840687 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 83419050085a486fbd270619d8aa2ae3 (DEFAULT_TABLE table=test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.840764 2630 raft_consensus.cc:374] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.840792 2630 raft_consensus.cc:729] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.840939 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83419050085a486fbd270619d8aa2ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.840907 2630 consensus_queue.cc:260] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.841152 2630 ts_tablet_manager.cc:1428] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:38:59.843428 2629 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Bootstrap starting.
I20260927 00:38:59.844437 2629 tablet_bootstrap.cc:654] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.844530 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 83419050085a486fbd270619d8aa2ae3 (DEFAULT_TABLE table=test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:38:59.844866 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83419050085a486fbd270619d8aa2ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:38:59.845595 2629 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: No bootstrap required, opened a new log
I20260927 00:38:59.845688 2629 ts_tablet_manager.cc:1397] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:38:59.846115 2629 raft_consensus.cc:348] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.846228 2629 raft_consensus.cc:374] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.846259 2629 raft_consensus.cc:729] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.846381 2629 consensus_queue.cc:260] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.846616 2629 ts_tablet_manager.cc:1428] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:38:59.848035 2628 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Bootstrap starting.
I20260927 00:38:59.849121 2628 tablet_bootstrap.cc:654] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Neither blocks nor log segments found. Creating new log.
I20260927 00:38:59.850441 2628 tablet_bootstrap.cc:492] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: No bootstrap required, opened a new log
I20260927 00:38:59.850538 2628 ts_tablet_manager.cc:1397] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:38:59.850960 2628 raft_consensus.cc:348] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.851063 2628 raft_consensus.cc:374] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:38:59.851090 2628 raft_consensus.cc:729] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Initialized, Role: FOLLOWER
I20260927 00:38:59.851125 2670 ts_tablet_manager.cc:1905] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:38:59.851224 2628 consensus_queue.cc:260] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.851500 2628 ts_tablet_manager.cc:1428] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:38:59.854584 2670 ts_tablet_manager.cc:1918] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:38:59.854718 2670 log.cc:1196] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/7066dc5d504c4ae99119d28783f50684
I20260927 00:38:59.855043 2670 ts_tablet_manager.cc:1939] T 7066dc5d504c4ae99119d28783f50684 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting consensus metadata
I20260927 00:38:59.856333 2108 catalog_manager.cc:4841] TS af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017): tablet 7066dc5d504c4ae99119d28783f50684 (table test-table [id=0551a3160d6b4f78801fec3ecbf69912]) successfully deleted
I20260927 00:38:59.971964 2635 raft_consensus.cc:482] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:59.972144 2635 raft_consensus.cc:504] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.973793 2635 leader_election.cc:288] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017)
I20260927 00:38:59.977819 2638 raft_consensus.cc:482] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:38:59.977949 2638 raft_consensus.cc:504] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.978415 2638 leader_election.cc:288] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:38:59.978699 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.973996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.2:40953)
0927 00:38:59.974676 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.974688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.974773 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.975008 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.975012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.975032 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.975164 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.975171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.976137 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.976148 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.977146 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.977159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.978026 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.978030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.978139 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.978165 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.978440 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.978518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":359,"thread_start_us":86,"threads_started":1}
I20260927 00:38:59.978858 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.974208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34303 (local address 127.0.98.1:42865)
0927 00:38:59.974334 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.974338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.974572 (+ 234us) server_negotiation.cc:407] Connection header received
0927 00:38:59.974705 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.974708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.974761 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.974854 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.975123 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.975912 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.977644 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.978222 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.978311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.978367 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.978664 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.978750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:38:59.979116 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd" is_pre_election: true
I20260927 00:38:59.979205 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.973996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42865 (local address 127.0.98.2:34303)
0927 00:38:59.974462 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 00:38:59.974476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:38:59.974605 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:38:59.974875 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:38:59.974878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:38:59.974890 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:38:59.975021 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.975028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.976052 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.976056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.977485 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:38:59.977498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.978423 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.978426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:38:59.978533 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.978554 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:38:59.978984 (+ 430us) client_negotiation.cc:240] Negotiation successful
0927 00:38:59.979057 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":36,"thread_start_us":193,"threads_started":1}
I20260927 00:38:59.979267 2270 raft_consensus.cc:2454] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.979348 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:38:59.974173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40953 (local address 127.0.98.3:36017)
0927 00:38:59.974294 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:38:59.974298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:38:59.974772 (+ 474us) server_negotiation.cc:407] Connection header received
0927 00:38:59.974851 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:38:59.974854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:38:59.974905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:38:59.974975 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:38:59.975292 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.976033 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.977277 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:38:59.977791 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:38:59.977894 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:38:59.977966 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:38:59.978287 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:38:59.978375 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:38:59.979512 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" is_pre_election: true
I20260927 00:38:59.979646 2419 raft_consensus.cc:2454] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:38:59.979893 2493 leader_election.cc:302] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:38:59.980157 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd" is_pre_election: true
I20260927 00:38:59.980315 2270 raft_consensus.cc:2454] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9060aa9e4e6649ed9c6b8f203da6a082 in term 0.
I20260927 00:38:59.980368 2638 raft_consensus.cc:2781] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:59.980448 2638 raft_consensus.cc:482] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:59.980484 2638 raft_consensus.cc:3037] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.980378 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" is_pre_election: true
I20260927 00:38:59.980700 2353 leader_election.cc:302] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:38:59.980988 2635 raft_consensus.cc:2781] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:38:59.981046 2635 raft_consensus.cc:482] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:38:59.981078 2635 raft_consensus.cc:3037] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.981621 2638 raft_consensus.cc:504] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.981938 2559 raft_consensus.cc:2379] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9060aa9e4e6649ed9c6b8f203da6a082 in current term 1: Already voted for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in this term.
I20260927 00:38:59.982031 2638 leader_election.cc:288] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Requested vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:38:59.982191 2635 raft_consensus.cc:504] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.982550 2635 leader_election.cc:288] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [CANDIDATE]: Term 1 election: Requested vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017)
I20260927 00:38:59.982936 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312"
I20260927 00:38:59.983063 2559 raft_consensus.cc:2379] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9060aa9e4e6649ed9c6b8f203da6a082 in current term 1: Already voted for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in this term.
I20260927 00:38:59.983265 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd"
I20260927 00:38:59.983383 2268 raft_consensus.cc:3037] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:38:59.984565 2268 raft_consensus.cc:2454] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9060aa9e4e6649ed9c6b8f203da6a082 in term 1.
I20260927 00:38:59.984802 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
I20260927 00:38:59.984880 2353 leader_election.cc:302] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, d6302ac49f5640ccaef46d26a3978ddd; no voters: af0d88bfc20e47a8b4e0a88f8fc59312
I20260927 00:38:59.984982 2419 raft_consensus.cc:2379] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af0d88bfc20e47a8b4e0a88f8fc59312 in current term 1: Already voted for candidate 9060aa9e4e6649ed9c6b8f203da6a082 in this term.
I20260927 00:38:59.985049 2635 raft_consensus.cc:2781] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:38:59.985265 2635 raft_consensus.cc:686] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 LEADER]: Becoming Leader. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Running, Role: LEADER
I20260927 00:38:59.985433 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83419050085a486fbd270619d8aa2ae3" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd"
I20260927 00:38:59.985602 2270 raft_consensus.cc:2379] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af0d88bfc20e47a8b4e0a88f8fc59312 in current term 1: Already voted for candidate 9060aa9e4e6649ed9c6b8f203da6a082 in this term.
I20260927 00:38:59.985535 2635 consensus_queue.cc:237] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:38:59.985950 2493 leader_election.cc:302] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312; no voters: 9060aa9e4e6649ed9c6b8f203da6a082, d6302ac49f5640ccaef46d26a3978ddd
I20260927 00:38:59.986291 2638 raft_consensus.cc:2726] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 00:38:59.988101 2119 catalog_manager.cc:5495] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:00.015462 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:00.012013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.0.98.2:36517)
0927 00:39:00.012252 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:00.012256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:00.012269 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:00.012339 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:00.012341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:00.012393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:00.012484 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:00.012726 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.013232 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.014073 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.014297 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.014383 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:00.014636 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:00.015079 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:00.015195 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:00.015264 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:39:00.015322 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:39:00.026508 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:00.022964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46907 (local address 127.0.98.3:36017)
0927 00:39:00.023076 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:00.023079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:00.023329 (+ 250us) server_negotiation.cc:407] Connection header received
0927 00:39:00.023420 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:00.023423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:00.023474 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:00.023538 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:00.023777 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.024410 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.025303 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.025527 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.025622 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:00.025976 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:00.026056 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:00.026194 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:00.026341 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:39:00.026390 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:39:00.026795 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:00.022897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.2:46907)
0927 00:39:00.023207 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:00.023220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:00.023318 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:00.023541 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:00.023543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:00.023548 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:00.023652 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:00.023656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.024520 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.024523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:00.025173 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:00.025185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.025673 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.025676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:00.025766 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:00.025833 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:00.026208 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:00.026215 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:39:00.026330 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:39:00.026409 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260927 00:39:00.032971 2270 raft_consensus.cc:1261] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Refusing update from remote peer 9060aa9e4e6649ed9c6b8f203da6a082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:00.033011 2559 raft_consensus.cc:1261] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Refusing update from remote peer 9060aa9e4e6649ed9c6b8f203da6a082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:00.033501 2677 consensus_queue.cc:1035] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:00.033679 2635 consensus_queue.cc:1035] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:00.051011 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:00.047228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36517 (local address 127.0.98.3:59885)
0927 00:39:00.047342 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:00.047358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:00.047431 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:00.047773 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:00.047777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:00.047785 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:00.047901 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:00.047907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.048880 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.048884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:00.049572 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:00.049585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.050273 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.050276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:00.050348 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:00.050379 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:00.050695 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:00.050703 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:39:00.050766 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:39:00.050861 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 00:39:00.051011 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:00.047309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59885 (local address 127.0.98.2:36517)
0927 00:39:00.047559 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:00.047565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:00.047579 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:00.047621 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:00.047623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:00.047668 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:00.047747 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:00.048004 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.048751 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.049719 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:00.049972 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:00.050077 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:00.050510 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:00.050605 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:00.050680 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:00.050796 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 00:39:00.050855 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
W20260927 00:39:00.051941 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:39:01.631762 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:01.631997 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:01.636036 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Sending DeleteTablet for 3 replicas of tablet 83419050085a486fbd270619d8aa2ae3
I20260927 00:39:01.636961 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 83419050085a486fbd270619d8aa2ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260927 00:39:01.636997 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 83419050085a486fbd270619d8aa2ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58420
I20260927 00:39:01.636997 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 83419050085a486fbd270619d8aa2ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57608
I20260927 00:39:01.637461 2708 tablet_replica.cc:330] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: stopping tablet replica
I20260927 00:39:01.637461 2709 tablet_replica.cc:330] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: stopping tablet replica
I20260927 00:39:01.637629 2709 raft_consensus.cc:2229] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:01.637629 2708 raft_consensus.cc:2229] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:01.637776 2709 raft_consensus.cc:2258] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.638015 2708 raft_consensus.cc:2258] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.638065 2707 tablet_replica.cc:330] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: stopping tablet replica
I20260927 00:39:01.638237 2707 raft_consensus.cc:2229] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:01.638365 2707 raft_consensus.cc:2258] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.639068 2709 ts_tablet_manager.cc:1905] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.640535 2707 ts_tablet_manager.cc:1905] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.641158 2708 ts_tablet_manager.cc:1905] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.642624 2709 ts_tablet_manager.cc:1918] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:39:01.642747 2709 log.cc:1196] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/83419050085a486fbd270619d8aa2ae3
I20260927 00:39:01.643041 2709 ts_tablet_manager.cc:1939] T 83419050085a486fbd270619d8aa2ae3 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting consensus metadata
I20260927 00:39:01.643879 2707 ts_tablet_manager.cc:1918] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:39:01.644044 2707 log.cc:1196] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/83419050085a486fbd270619d8aa2ae3
I20260927 00:39:01.644362 2707 ts_tablet_manager.cc:1939] T 83419050085a486fbd270619d8aa2ae3 P d6302ac49f5640ccaef46d26a3978ddd: Deleting consensus metadata
I20260927 00:39:01.644388 2708 ts_tablet_manager.cc:1918] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:39:01.644492 2708 log.cc:1196] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/83419050085a486fbd270619d8aa2ae3
I20260927 00:39:01.644600 2108 catalog_manager.cc:4841] TS af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017): tablet 83419050085a486fbd270619d8aa2ae3 (table test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]) successfully deleted
I20260927 00:39:01.644867 2708 ts_tablet_manager.cc:1939] T 83419050085a486fbd270619d8aa2ae3 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting consensus metadata
I20260927 00:39:01.645231 2109 catalog_manager.cc:4841] TS d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865): tablet 83419050085a486fbd270619d8aa2ae3 (table test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]) successfully deleted
I20260927 00:39:01.645810 2108 catalog_manager.cc:4841] TS 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517): tablet 83419050085a486fbd270619d8aa2ae3 (table test-table [id=c47e7b44c43a41d7a6a8ae8bd19c4826]) successfully deleted
Sep 27 00:39:01 dist-test-slave-xltq krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:39:01 dist-test-slave-xltq krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:01.668866 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.659223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.62:44131)
0927 00:39:01.659576 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.659585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.659602 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:39:01.659747 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.659750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.659798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.659902 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:01.660798 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.661413 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.662357 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.662619 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.662709 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.665419 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:01.665423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:01.665425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:01.665456 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:01.667110 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:01.667623 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:01.667626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:01.667628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:01.667713 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:01.667984 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:01.667987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:01.667989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:01.668182 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:01.668330 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.668557 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.668684 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260927 00:39:01.672145 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:01.672631 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:01.677999 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (DEFAULT_TABLE table=test-table [id=8a4da710640244eab21e7dae64936930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.677976 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (DEFAULT_TABLE table=test-table [id=8a4da710640244eab21e7dae64936930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.678103 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (DEFAULT_TABLE table=test-table [id=8a4da710640244eab21e7dae64936930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.678328 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.678339 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.678359 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.680989 2719 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Bootstrap starting.
I20260927 00:39:01.681291 2718 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Bootstrap starting.
I20260927 00:39:01.682047 2719 tablet_bootstrap.cc:654] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.682221 2718 tablet_bootstrap.cc:654] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.682794 2720 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Bootstrap starting.
I20260927 00:39:01.683626 2719 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: No bootstrap required, opened a new log
I20260927 00:39:01.683732 2719 ts_tablet_manager.cc:1397] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:01.683823 2720 tablet_bootstrap.cc:654] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.684238 2719 raft_consensus.cc:348] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.684392 2719 raft_consensus.cc:374] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.684435 2719 raft_consensus.cc:729] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.684597 2719 consensus_queue.cc:260] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.684654 2718 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: No bootstrap required, opened a new log
I20260927 00:39:01.684746 2718 ts_tablet_manager.cc:1397] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:39:01.684931 2719 ts_tablet_manager.cc:1428] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:01.685202 2718 raft_consensus.cc:348] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.685302 2718 raft_consensus.cc:374] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.685308 2720 tablet_bootstrap.cc:492] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: No bootstrap required, opened a new log
I20260927 00:39:01.685330 2718 raft_consensus.cc:729] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.685396 2720 ts_tablet_manager.cc:1397] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:01.685452 2718 consensus_queue.cc:260] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.685850 2718 ts_tablet_manager.cc:1428] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:01.685813 2720 raft_consensus.cc:348] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.685926 2720 raft_consensus.cc:374] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.685961 2720 raft_consensus.cc:729] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.686091 2720 consensus_queue.cc:260] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.686380 2720 ts_tablet_manager.cc:1428] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:39:01.723688 2667 raft_consensus.cc:482] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:01.723842 2667 raft_consensus.cc:504] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.724298 2667 leader_election.cc:288] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:39:01.724786 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c3acdd7c04592b9e21ddf8f1c3d3b" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd" is_pre_election: true
I20260927 00:39:01.724821 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c3acdd7c04592b9e21ddf8f1c3d3b" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" is_pre_election: true
I20260927 00:39:01.724969 2419 raft_consensus.cc:2454] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:39:01.724974 2268 raft_consensus.cc:2454] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 0.
I20260927 00:39:01.725296 2493 leader_election.cc:302] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:39:01.725518 2667 raft_consensus.cc:2781] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:01.725584 2667 raft_consensus.cc:482] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:01.725630 2667 raft_consensus.cc:3037] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.726665 2667 raft_consensus.cc:504] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.727031 2667 leader_election.cc:288] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Requested vote from peers d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865), 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517)
I20260927 00:39:01.727483 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c3acdd7c04592b9e21ddf8f1c3d3b" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6302ac49f5640ccaef46d26a3978ddd"
I20260927 00:39:01.727500 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096c3acdd7c04592b9e21ddf8f1c3d3b" candidate_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
I20260927 00:39:01.727634 2268 raft_consensus.cc:3037] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.727634 2419 raft_consensus.cc:3037] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.728843 2268 raft_consensus.cc:2454] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:39:01.728843 2419 raft_consensus.cc:2454] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d88bfc20e47a8b4e0a88f8fc59312 in term 1.
I20260927 00:39:01.729231 2492 leader_election.cc:302] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:39:01.729414 2667 raft_consensus.cc:2781] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:01.729508 2667 raft_consensus.cc:686] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 LEADER]: Becoming Leader. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Running, Role: LEADER
I20260927 00:39:01.729656 2667 consensus_queue.cc:237] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.731508 2119 catalog_manager.cc:5495] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 reported cstate change: term changed from 0 to 1, leader changed from <none> to af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: HEALTHY } } }
I20260927 00:39:01.759423 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.755970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.98.3:36017)
0927 00:39:01.756240 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.756249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.756263 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:01.756312 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.756315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.756374 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.756451 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:01.756730 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.757368 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.758326 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.758612 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.758691 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.758961 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.759034 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.759087 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.759228 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.759277 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260927 00:39:01.766281 2419 raft_consensus.cc:1261] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:01.766281 2268 raft_consensus.cc:1261] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Refusing update from remote peer af0d88bfc20e47a8b4e0a88f8fc59312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:01.766703 2693 consensus_queue.cc:1035] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:01.766839 2667 consensus_queue.cc:1035] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:01.807085 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:01.807229 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:01.809258 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Sending DeleteTablet for 3 replicas of tablet 096c3acdd7c04592b9e21ddf8f1c3d3b
I20260927 00:39:01.809891 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260927 00:39:01.809934 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58420
I20260927 00:39:01.809959 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 096c3acdd7c04592b9e21ddf8f1c3d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57608
I20260927 00:39:01.810120 2709 tablet_replica.cc:330] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: stopping tablet replica
I20260927 00:39:01.810124 2707 tablet_replica.cc:330] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: stopping tablet replica
I20260927 00:39:01.810127 2708 tablet_replica.cc:330] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: stopping tablet replica
I20260927 00:39:01.810245 2709 raft_consensus.cc:2229] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:01.810243 2707 raft_consensus.cc:2229] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:01.810245 2708 raft_consensus.cc:2229] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:01.810370 2707 raft_consensus.cc:2258] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.810370 2708 raft_consensus.cc:2258] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.810437 2709 raft_consensus.cc:2258] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:01.811100 2708 ts_tablet_manager.cc:1905] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.811462 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:01.811928 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:01.812615 2707 ts_tablet_manager.cc:1905] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.814062 2708 ts_tablet_manager.cc:1918] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:01.814149 2708 log.cc:1196] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/096c3acdd7c04592b9e21ddf8f1c3d3b
I20260927 00:39:01.814424 2708 ts_tablet_manager.cc:1939] T 096c3acdd7c04592b9e21ddf8f1c3d3b P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting consensus metadata
I20260927 00:39:01.815124 2707 ts_tablet_manager.cc:1918] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:01.815197 2707 log.cc:1196] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/096c3acdd7c04592b9e21ddf8f1c3d3b
I20260927 00:39:01.815215 2108 catalog_manager.cc:4841] TS 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517): tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (table test-table [id=8a4da710640244eab21e7dae64936930]) successfully deleted
I20260927 00:39:01.815426 2707 ts_tablet_manager.cc:1939] T 096c3acdd7c04592b9e21ddf8f1c3d3b P d6302ac49f5640ccaef46d26a3978ddd: Deleting consensus metadata
I20260927 00:39:01.816247 2709 ts_tablet_manager.cc:1905] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:01.817621 2109 catalog_manager.cc:4841] TS d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865): tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (table test-table [id=8a4da710640244eab21e7dae64936930]) successfully deleted
I20260927 00:39:01.817687 2399 tablet_service.cc:1467] Processing CreateTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 (DEFAULT_TABLE table=test-table [id=4da618fb9af5453e8d99a3ee1b6ea9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.817951 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3776c88c0c14156aa06629b8bbdeec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.818801 2709 ts_tablet_manager.cc:1918] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:01.818874 2709 log.cc:1196] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/096c3acdd7c04592b9e21ddf8f1c3d3b
I20260927 00:39:01.819077 2709 ts_tablet_manager.cc:1939] T 096c3acdd7c04592b9e21ddf8f1c3d3b P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting consensus metadata
I20260927 00:39:01.819717 2108 catalog_manager.cc:4841] TS af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017): tablet 096c3acdd7c04592b9e21ddf8f1c3d3b (table test-table [id=8a4da710640244eab21e7dae64936930]) successfully deleted
I20260927 00:39:01.819816 2539 tablet_service.cc:1467] Processing CreateTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 (DEFAULT_TABLE table=test-table [id=4da618fb9af5453e8d99a3ee1b6ea9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.820039 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3776c88c0c14156aa06629b8bbdeec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.820080 2719 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Bootstrap starting.
I20260927 00:39:01.820353 2259 tablet_service.cc:1467] Processing CreateTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 (DEFAULT_TABLE table=test-table [id=4da618fb9af5453e8d99a3ee1b6ea9f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:01.820621 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3776c88c0c14156aa06629b8bbdeec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:01.821044 2719 tablet_bootstrap.cc:654] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.821944 2720 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Bootstrap starting.
I20260927 00:39:01.822173 2719 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: No bootstrap required, opened a new log
I20260927 00:39:01.822252 2719 ts_tablet_manager.cc:1397] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:01.822755 2720 tablet_bootstrap.cc:654] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.822961 2718 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: Bootstrap starting.
I20260927 00:39:01.823753 2720 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: No bootstrap required, opened a new log
I20260927 00:39:01.822698 2719 raft_consensus.cc:348] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.823843 2720 ts_tablet_manager.cc:1397] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:39:01.823877 2719 raft_consensus.cc:374] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.823917 2719 raft_consensus.cc:729] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9060aa9e4e6649ed9c6b8f203da6a082, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.823947 2718 tablet_bootstrap.cc:654] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:01.824060 2719 consensus_queue.cc:260] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.824313 2720 raft_consensus.cc:348] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.824384 2719 ts_tablet_manager.cc:1428] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:39:01.824410 2720 raft_consensus.cc:374] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.824441 2720 raft_consensus.cc:729] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d88bfc20e47a8b4e0a88f8fc59312, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.824559 2720 consensus_queue.cc:260] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.824831 2720 ts_tablet_manager.cc:1428] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:01.825146 2718 tablet_bootstrap.cc:492] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: No bootstrap required, opened a new log
I20260927 00:39:01.825227 2718 ts_tablet_manager.cc:1397] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:01.825659 2718 raft_consensus.cc:348] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.825774 2718 raft_consensus.cc:374] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:01.825819 2718 raft_consensus.cc:729] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Initialized, Role: FOLLOWER
I20260927 00:39:01.825939 2718 consensus_queue.cc:260] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.826210 2718 ts_tablet_manager.cc:1428] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:39:01.885426 2724 raft_consensus.cc:482] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:01.885571 2724 raft_consensus.cc:504] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.886662 2724 leader_election.cc:288] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517), af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017)
I20260927 00:39:01.890556 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.887020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46847 (local address 127.0.98.2:36517)
0927 00:39:01.887258 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.887271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.887289 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:01.887460 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.887463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.887516 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.887601 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:01.887877 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.888444 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.889558 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.889986 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.890082 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.890150 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.890313 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.890405 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260927 00:39:01.890576 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.886910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36517 (local address 127.0.98.1:46847)
0927 00:39:01.887176 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:01.887190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:01.887336 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:01.887623 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:01.887626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:01.887659 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:01.887771 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.887776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.888548 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.888551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.889440 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.889449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.890098 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.890102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.890202 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.890224 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:39:01.890291 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:39:01.890382 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":4,"spinlock_wait_cycles":6912,"thread_start_us":85,"threads_started":1}
I20260927 00:39:01.890664 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.887579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33201 (local address 127.0.98.3:36017)
0927 00:39:01.887690 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.887694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.887707 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:01.887756 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.887758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.887800 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.887870 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:01.888128 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.888711 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.889734 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.890144 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.890236 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.890298 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.890471 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.890548 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:39:01.890895 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.887258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.1:33201)
0927 00:39:01.887559 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:01.887571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:01.887656 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:01.887902 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:01.887905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:01.887921 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:01.888033 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.888038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.888835 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.888838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.889612 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.889621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.890251 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.890254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.890362 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.890382 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:01.890447 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:39:01.890517 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":25,"spinlock_wait_cycles":2944,"thread_start_us":108,"threads_started":1}
I20260927 00:39:01.891041 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3776c88c0c14156aa06629b8bbdeec2" candidate_uuid: "d6302ac49f5640ccaef46d26a3978ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" is_pre_election: true
I20260927 00:39:01.891188 2419 raft_consensus.cc:2454] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6302ac49f5640ccaef46d26a3978ddd in term 0.
I20260927 00:39:01.891479 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3776c88c0c14156aa06629b8bbdeec2" candidate_uuid: "d6302ac49f5640ccaef46d26a3978ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" is_pre_election: true
I20260927 00:39:01.891629 2559 raft_consensus.cc:2454] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6302ac49f5640ccaef46d26a3978ddd in term 0.
I20260927 00:39:01.891577 2212 leader_election.cc:302] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9060aa9e4e6649ed9c6b8f203da6a082, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:39:01.891857 2724 raft_consensus.cc:2781] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:01.891918 2724 raft_consensus.cc:482] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:01.891958 2724 raft_consensus.cc:3037] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.893060 2724 raft_consensus.cc:504] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.893407 2724 leader_election.cc:288] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [CANDIDATE]: Term 1 election: Requested vote from peers 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517), af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017)
I20260927 00:39:01.893757 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3776c88c0c14156aa06629b8bbdeec2" candidate_uuid: "d6302ac49f5640ccaef46d26a3978ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9060aa9e4e6649ed9c6b8f203da6a082"
I20260927 00:39:01.893760 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3776c88c0c14156aa06629b8bbdeec2" candidate_uuid: "d6302ac49f5640ccaef46d26a3978ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312"
I20260927 00:39:01.893868 2419 raft_consensus.cc:3037] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.893872 2559 raft_consensus.cc:3037] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:01.894833 2559 raft_consensus.cc:2454] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6302ac49f5640ccaef46d26a3978ddd in term 1.
I20260927 00:39:01.894870 2419 raft_consensus.cc:2454] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6302ac49f5640ccaef46d26a3978ddd in term 1.
I20260927 00:39:01.895112 2212 leader_election.cc:302] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d88bfc20e47a8b4e0a88f8fc59312, d6302ac49f5640ccaef46d26a3978ddd; no voters:
I20260927 00:39:01.895268 2724 raft_consensus.cc:2781] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:01.895505 2724 raft_consensus.cc:686] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 1 LEADER]: Becoming Leader. State: Replica: d6302ac49f5640ccaef46d26a3978ddd, State: Running, Role: LEADER
I20260927 00:39:01.895750 2724 consensus_queue.cc:237] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } }
I20260927 00:39:01.898053 2118 catalog_manager.cc:5495] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d6302ac49f5640ccaef46d26a3978ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6302ac49f5640ccaef46d26a3978ddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:01.921784 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.917583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.0.98.1:42865)
0927 00:39:01.917854 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.917862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.917874 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:39:01.917957 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.917959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.918012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.918115 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:01.918352 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.919338 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.920181 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.920446 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.920538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.920805 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.921328 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.921422 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.921584 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.921635 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
I20260927 00:39:01.933061 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.929160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36017 (local address 127.0.98.1:42107)
0927 00:39:01.929559 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:01.929572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:01.929675 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:01.929980 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:01.929982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:01.929991 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:01.930116 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.930120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.930966 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.930969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.931623 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.931633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.932168 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.932171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.932304 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.932359 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.932734 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.932741 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:39:01.932812 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:39:01.932892 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":89,"threads_started":1}
I20260927 00:39:01.933095 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.929301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42107 (local address 127.0.98.3:36017)
0927 00:39:01.929562 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.929566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.929642 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:39:01.929799 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.929802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.929855 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.929927 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:01.930231 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.930821 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.931755 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.932006 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.932096 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.932541 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.932630 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.932688 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.932928 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.932992 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 00:39:01.938957 2419 raft_consensus.cc:1261] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Refusing update from remote peer d6302ac49f5640ccaef46d26a3978ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:01.938969 2559 raft_consensus.cc:1261] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Refusing update from remote peer d6302ac49f5640ccaef46d26a3978ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:01.939388 2724 consensus_queue.cc:1035] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d88bfc20e47a8b4e0a88f8fc59312" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:01.939589 2737 consensus_queue.cc:1035] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9060aa9e4e6649ed9c6b8f203da6a082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:01.957305 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.953439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42865 (local address 127.0.98.3:34649)
0927 00:39:01.953707 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:01.953719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:01.953806 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:01.954154 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:01.954156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:01.954165 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:01.954262 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.954267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.955108 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.955111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.955829 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:01.955840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.956561 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.956564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:01.956630 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.956665 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.957001 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.957009 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:39:01.957076 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:39:01.957179 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260927 00:39:01.957286 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:01.953507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34649 (local address 127.0.98.1:42865)
0927 00:39:01.953934 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:01.953937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:01.953950 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:01.953998 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:01.954000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:01.954042 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:01.954104 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:01.954402 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.954991 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.955941 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:01.956353 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:01.956551 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:01.956784 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:01.956914 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:01.956983 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:01.957092 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:39:01.957147 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
W20260927 00:39:01.958212 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:39:03.520303 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:03.520550 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:03.523789 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b87175b2b9445ed890810cc0bc0b09d: Sending DeleteTablet for 3 replicas of tablet f3776c88c0c14156aa06629b8bbdeec2
I20260927 00:39:03.524662 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57608
I20260927 00:39:03.524760 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260927 00:39:03.524816 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet f3776c88c0c14156aa06629b8bbdeec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58420
I20260927 00:39:03.525148 2771 tablet_replica.cc:330] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd: stopping tablet replica
I20260927 00:39:03.525301 2771 raft_consensus.cc:2229] T f3776c88c0c14156aa06629b8bbdeec2 P d6302ac49f5640ccaef46d26a3978ddd [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:03.525316 2772 tablet_replica.cc:330] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: stopping tablet replica
I20260927 00:39:03.525417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2194
I20260927 00:39:03.525488 2772 raft_consensus.cc:2229] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:03.526255 2770 tablet_replica.cc:330] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: stopping tablet replica
I20260927 00:39:03.526372 2772 raft_consensus.cc:2258] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:03.526399 2770 raft_consensus.cc:2229] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:03.526531 2770 raft_consensus.cc:2258] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:03.527221 2770 ts_tablet_manager.cc:1905] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:03.529284 2772 ts_tablet_manager.cc:1905] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:03.529596 2770 ts_tablet_manager.cc:1918] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:39:03.529671 2770 log.cc:1196] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/f3776c88c0c14156aa06629b8bbdeec2
I20260927 00:39:03.529938 2770 ts_tablet_manager.cc:1939] T f3776c88c0c14156aa06629b8bbdeec2 P 9060aa9e4e6649ed9c6b8f203da6a082: Deleting consensus metadata
I20260927 00:39:03.530678 2108 catalog_manager.cc:4841] TS 9060aa9e4e6649ed9c6b8f203da6a082 (127.0.98.2:36517): tablet f3776c88c0c14156aa06629b8bbdeec2 (table test-table [id=4da618fb9af5453e8d99a3ee1b6ea9f0]) successfully deleted
I20260927 00:39:03.532716 2772 ts_tablet_manager.cc:1918] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:39:03.532829 2772 log.cc:1196] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/f3776c88c0c14156aa06629b8bbdeec2
I20260927 00:39:03.533090 2772 ts_tablet_manager.cc:1939] T f3776c88c0c14156aa06629b8bbdeec2 P af0d88bfc20e47a8b4e0a88f8fc59312: Deleting consensus metadata
I20260927 00:39:03.534168 2108 catalog_manager.cc:4841] TS af0d88bfc20e47a8b4e0a88f8fc59312 (127.0.98.3:36017): tablet f3776c88c0c14156aa06629b8bbdeec2 (table test-table [id=4da618fb9af5453e8d99a3ee1b6ea9f0]) successfully deleted
W20260927 00:39:03.535521 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42865 (error 108)
I20260927 00:39:03.536523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2334
I20260927 00:39:03.537686 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:03.536509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42865 (local address 127.0.0.1:36926)
0927 00:39:03.537437 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:03.537546 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":116,"threads_started":1}
W20260927 00:39:03.537833 2109 catalog_manager.cc:4584] TS d6302ac49f5640ccaef46d26a3978ddd (127.0.98.1:42865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3776c88c0c14156aa06629b8bbdeec2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42865: connect: Connection refused (error 111)
I20260927 00:39:03.543062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2474
I20260927 00:39:03.549854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2090
2026-09-27T00:39:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6249 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:03 dist-test-slave-xltq krb5kdc[2777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:03 dist-test-slave-xltq krb5kdc[2777](info): set up 2 sockets
Sep 27 00:39:03 dist-test-slave-xltq krb5kdc[2777](info): commencing operation
krb5kdc: starting...
W20260927 00:39:04.523984 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:04 dist-test-slave-xltq krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:04.647087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:04.758714 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:04.759020 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:04.759076 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:04.763250 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:04.763330 2790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:04.763356 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:04.763377 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:04.763397 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:04.769042 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:04.770532 2790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:04.771797 2790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:04.780854 2795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:04.780927 2796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:04.780850 2798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:04.781343 2790 server_base.cc:1034] running on GCE node
I20260927 00:39:04.781919 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:04.783130 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:04.784291 2790 hybrid_clock.cc:648] HybridClock initialized: now 1790469544784269 us; error 37 us; skew 500 ppm
Sep 27 00:39:04 dist-test-slave-xltq krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:04.787384 2790 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:04.788503 2790 webserver.cc:460] Webserver started at http://127.0.98.62:37651/ using document root <none> and password file <none>
I20260927 00:39:04.789156 2790 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:04.789234 2790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:04.789455 2790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:04.791893 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "e08a9a113c4a4e858a97f8c3e4c08c81"
format_stamp: "Formatted at 2026-09-27 00:39:04 on dist-test-slave-xltq"
server_key: "c6e849294ac78c758abb94e967560690"
server_key_iv: "e3a4ce73d0320dee92b9f5043dc41aaa"
server_key_version: "encryptionkey@0"
I20260927 00:39:04.792518 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "e08a9a113c4a4e858a97f8c3e4c08c81"
format_stamp: "Formatted at 2026-09-27 00:39:04 on dist-test-slave-xltq"
server_key: "c6e849294ac78c758abb94e967560690"
server_key_iv: "e3a4ce73d0320dee92b9f5043dc41aaa"
server_key_version: "encryptionkey@0"
I20260927 00:39:04.796761 2790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:04.799463 2805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:04.800916 2790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:04.801044 2790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "e08a9a113c4a4e858a97f8c3e4c08c81"
format_stamp: "Formatted at 2026-09-27 00:39:04 on dist-test-slave-xltq"
server_key: "c6e849294ac78c758abb94e967560690"
server_key_iv: "e3a4ce73d0320dee92b9f5043dc41aaa"
server_key_version: "encryptionkey@0"
I20260927 00:39:04.801155 2790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:04.827072 2790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:04.830734 2790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:04.830938 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:04.837929 2790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41515
I20260927 00:39:04.838013 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41515 every 8 connection(s)
I20260927 00:39:04.838913 2790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:04.841871 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:04.843706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2790
I20260927 00:39:04.843792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:04.844020 392 external_mini_cluster.cc:1428] Setting key ecc2630360eda65fa091bec34d7c2cba
I20260927 00:39:04.848392 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81: Bootstrap starting.
I20260927 00:39:04.850574 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:04.851266 2858 log.cc:824] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:39:04 dist-test-slave-xltq krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:04.853277 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81: No bootstrap required, opened a new log
I20260927 00:39:04.857656 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } }
I20260927 00:39:04.857961 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:04.858007 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a9a113c4a4e858a97f8c3e4c08c81, State: Initialized, Role: FOLLOWER
I20260927 00:39:04.858395 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:04.845494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.98.62:41515)
0927 00:39:04.845948 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:04.845971 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:04.846008 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:39:04.846661 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:04.846679 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:04.847023 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:04.847385 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:04.848115 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:04.849157 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:04.849964 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:04.850434 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:04.850573 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:04.852866 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:04.852872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:04.852889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:04.852918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:04.855051 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:04.855479 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:04.855486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:04.855493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:04.855643 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:04.855961 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:04.855964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:04.855966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:04.856515 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:04.856741 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:04.857030 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:39:04.857351 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260927 00:39:04.858484 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } }
I20260927 00:39:04.858632 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:04.858698 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:04.858793 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:04.859898 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } }
I20260927 00:39:04.860267 2858 leader_election.cc:302] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e08a9a113c4a4e858a97f8c3e4c08c81; no voters:
I20260927 00:39:04.860596 2858 leader_election.cc:288] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:04.860702 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:04.860913 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [term 1 LEADER]: Becoming Leader. State: Replica: e08a9a113c4a4e858a97f8c3e4c08c81, State: Running, Role: LEADER
I20260927 00:39:04.861233 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } }
I20260927 00:39:04.861810 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:04.862802 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } } }
I20260927 00:39:04.862931 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:04.863188 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e08a9a113c4a4e858a97f8c3e4c08c81. Latest consensus state: current_term: 1 leader_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e08a9a113c4a4e858a97f8c3e4c08c81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41515 } } }
I20260927 00:39:04.863287 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:04.863399 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:04.866369 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:04.871802 2872 catalog_manager.cc:1347] Generated new cluster ID: 436f9345148e481aaaae96cbf043888a
I20260927 00:39:04.871878 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:04.882524 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:04.883494 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:04.889106 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e08a9a113c4a4e858a97f8c3e4c08c81: Generated new TSK 0
I20260927 00:39:04.889958 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:04.921612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:05.034682 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:05.035005 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:05.035053 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:05.039409 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:05.039495 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:05.039558 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:05.045572 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:05.046720 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:05.047912 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:05.058161 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.058166 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.058192 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:05.058561 2884 server_base.cc:1034] running on GCE node
I20260927 00:39:05.058981 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:05.059687 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:05.060895 2884 hybrid_clock.cc:648] HybridClock initialized: now 1790469545060858 us; error 37 us; skew 500 ppm
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:05.064002 2884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:05.064987 2884 webserver.cc:460] Webserver started at http://127.0.98.1:35827/ using document root <none> and password file <none>
I20260927 00:39:05.065568 2884 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:05.065631 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:05.065816 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:05.067940 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "7929c5b486764a1694c85af33ede0acf"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0eeb0160d3b1a4ab0e552d3ded8910c9"
server_key_iv: "1525c6623d3b891d63db6e79b2eb7de3"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.068549 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7929c5b486764a1694c85af33ede0acf"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0eeb0160d3b1a4ab0e552d3ded8910c9"
server_key_iv: "1525c6623d3b891d63db6e79b2eb7de3"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.072623 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:05.075119 2899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.076470 2884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:39:05.076575 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "7929c5b486764a1694c85af33ede0acf"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0eeb0160d3b1a4ab0e552d3ded8910c9"
server_key_iv: "1525c6623d3b891d63db6e79b2eb7de3"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.076680 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:05.095639 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:05.098701 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:05.098872 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:05.099453 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:05.100303 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:05.100356 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.100400 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:05.100427 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.108903 2884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34381
I20260927 00:39:05.108942 3012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34381 every 8 connection(s)
I20260927 00:39:05.109802 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:05.118183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 2884
I20260927 00:39:05.118291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:05.118561 392 external_mini_cluster.cc:1428] Setting key 24c12b4af99b8e81247f0717c7a33ae3
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:05.124361 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.111684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55523 (local address 127.0.98.62:41515)
0927 00:39:05.111866 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:05.111875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:05.112731 (+ 856us) server_negotiation.cc:407] Connection header received
0927 00:39:05.114208 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:05.114212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:05.114278 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:05.114352 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:05.115598 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.116115 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.117268 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.117497 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.117587 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.120223 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:05.120227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:05.120231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:05.120258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:05.121907 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.122624 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.122627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.122628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.122727 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.123145 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.123148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.123150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.123352 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:05.123441 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:05.124003 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 00:39:05.124164 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:39:05.125216 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.112064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41515 (local address 127.0.98.1:55523)
0927 00:39:05.112571 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:05.112615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:05.113986 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:05.114545 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:05.114552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:05.114941 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:05.115448 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.115462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.116252 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.116255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.117141 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.117151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.117642 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.117647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.117812 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.118402 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:05.118425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:05.120037 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:05.122046 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.122054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.122074 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.122496 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.122854 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.122857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.122859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.123050 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.123456 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:05.123475 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:05.123746 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 00:39:05.123964 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:39:05.124347 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":156,"threads_started":1}
I20260927 00:39:05.126554 3013 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41515
I20260927 00:39:05.126811 3013 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:05.127388 3013 heartbeater.cc:511] Master 127.0.98.62:41515 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:05.129065 2822 ts_manager.cc:194] Registered new tserver with Master: 7929c5b486764a1694c85af33ede0acf (127.0.98.1:34381)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:05.130611 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:05.146903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:05.257130 3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:05.257400 3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:05.257447 3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:05.261955 3018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:05.262040 3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:05.262116 3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:05.267930 3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:05.269085 3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:05.270133 3018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:05.280444 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.280498 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.280539 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:05.280793 3018 server_base.cc:1034] running on GCE node
I20260927 00:39:05.281306 3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:05.282029 3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:05.283211 3018 hybrid_clock.cc:648] HybridClock initialized: now 1790469545283186 us; error 36 us; skew 500 ppm
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:05.286234 3018 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:05.287251 3018 webserver.cc:460] Webserver started at http://127.0.98.2:46113/ using document root <none> and password file <none>
I20260927 00:39:05.287815 3018 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:05.287878 3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:05.288075 3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:05.290369 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8dd65b75a9a47a3aa3c501feda138e3"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0dedb263c8a8a823e0aea1bd7d575fd4"
server_key_iv: "b53718bd736b16fb18e59343a5e3d8ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.291075 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8dd65b75a9a47a3aa3c501feda138e3"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0dedb263c8a8a823e0aea1bd7d575fd4"
server_key_iv: "b53718bd736b16fb18e59343a5e3d8ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.295298 3018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:39:05.298429 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.300052 3018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:05.300227 3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "a8dd65b75a9a47a3aa3c501feda138e3"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "0dedb263c8a8a823e0aea1bd7d575fd4"
server_key_iv: "b53718bd736b16fb18e59343a5e3d8ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.300359 3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:05.331868 3018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:05.335675 3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:05.335907 3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:05.336599 3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:05.337563 3018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:05.337620 3018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.337682 3018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:05.337713 3018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.347615 3018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33351
I20260927 00:39:05.347645 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33351 every 8 connection(s)
I20260927 00:39:05.348726 3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:05.353835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3018
I20260927 00:39:05.353956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:05.354238 392 external_mini_cluster.cc:1428] Setting key 27c79849e2828209ca848b97577d75fe
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:05.365537 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.351009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42723 (local address 127.0.98.62:41515)
0927 00:39:05.351173 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:05.351177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:05.352135 (+ 958us) server_negotiation.cc:407] Connection header received
0927 00:39:05.353631 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:05.353634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:05.353699 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:05.353771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:05.355245 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.355865 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.357091 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.357369 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.357486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.360441 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:05.360445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:05.360448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:05.360483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:05.362408 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.363234 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.363238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.363240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.363359 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.363867 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.363870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.363872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.364137 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:05.364316 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:05.365180 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 00:39:05.365320 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:39:05.366410 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.351312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41515 (local address 127.0.98.2:42723)
0927 00:39:05.351952 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:05.352002 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:05.353368 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:05.353991 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:05.354003 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:05.354450 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:05.355084 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.355102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.355990 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.355994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.356953 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.356964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.357558 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.357562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.357781 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.358500 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:05.358519 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:05.360164 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:05.362632 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.362660 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.362679 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.363100 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.363513 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.363516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.363517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.363757 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.364411 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:05.364418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:05.364748 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 00:39:05.365039 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 00:39:05.365430 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260927 00:39:05.367956 3147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:05.368289 3147 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:05.368875 3147 heartbeater.cc:511] Master 127.0.98.62:41515 requested a full tablet report, sending...
I20260927 00:39:05.370216 2822 ts_manager.cc:194] Registered new tserver with Master: a8dd65b75a9a47a3aa3c501feda138e3 (127.0.98.2:33351)
I20260927 00:39:05.370951 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:05.388314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:05.503312 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:05.503644 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:05.503690 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:05.508041 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:05.508111 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:05.508172 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:05.513623 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:05.514683 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:05.515657 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:05.525585 3152 server_base.cc:1034] running on GCE node
W20260927 00:39:05.525528 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.525525 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:05.525545 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:05.526187 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:05.526893 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:05.528065 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790469545528032 us; error 53 us; skew 500 ppm
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:05.530951 3152 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:05.531899 3152 webserver.cc:460] Webserver started at http://127.0.98.3:40177/ using document root <none> and password file <none>
I20260927 00:39:05.532496 3152 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:05.532562 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:05.532755 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:05.534814 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "6908d321add442389cf4964e10937f35"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "661ea67a0758d56ff1bfcf2c90c21c16"
server_key_iv: "a99265be7bf8fed683a57094fe8766d0"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.535359 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6908d321add442389cf4964e10937f35"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "661ea67a0758d56ff1bfcf2c90c21c16"
server_key_iv: "a99265be7bf8fed683a57094fe8766d0"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.539270 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:39:05.542100 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.543570 3152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:05.543747 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "6908d321add442389cf4964e10937f35"
format_stamp: "Formatted at 2026-09-27 00:39:05 on dist-test-slave-xltq"
server_key: "661ea67a0758d56ff1bfcf2c90c21c16"
server_key_iv: "a99265be7bf8fed683a57094fe8766d0"
server_key_version: "encryptionkey@0"
I20260927 00:39:05.543955 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:05.570506 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:05.574249 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:05.574447 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:05.575096 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:05.576013 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:05.576067 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.576109 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:05.576130 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:05.585664 3152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41507
I20260927 00:39:05.585695 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41507 every 8 connection(s)
I20260927 00:39:05.586634 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:05.595734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3152
I20260927 00:39:05.595841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:05.596127 392 external_mini_cluster.cc:1428] Setting key 4c348c502d72ff45db95e506bae8363c
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:05.601941 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.588613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60477 (local address 127.0.98.62:41515)
0927 00:39:05.588801 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:05.588806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:05.589797 (+ 991us) server_negotiation.cc:407] Connection header received
0927 00:39:05.591340 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:05.591344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:05.591405 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:05.591508 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:05.593003 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.593536 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.594940 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.595240 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.595332 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.598349 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:05.598353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:05.598356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:05.598385 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:05.599776 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.600358 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.600362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.600363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.600454 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:05.600796 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:05.600799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:05.600801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:05.601003 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:05.601088 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:05.601644 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 00:39:05.601773 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 00:39:05.602807 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:05.589071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41515 (local address 127.0.98.3:60477)
0927 00:39:05.589620 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:05.589672 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:05.591066 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:05.591815 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:05.591824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:05.592281 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:05.592828 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.592843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.593686 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.593689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.594756 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:05.594768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:05.595424 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:05.595428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:05.595606 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:05.596362 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:05.596385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:05.598171 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:05.599906 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.599922 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.599943 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.600262 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.600537 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:05.600541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:05.600543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:05.600700 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:05.601102 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:05.601110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:05.601402 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 00:39:05.601578 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:39:05.601963 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":149,"threads_started":1}
I20260927 00:39:05.604187 3281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41515
I20260927 00:39:05.604583 3281 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:05.605247 3281 heartbeater.cc:511] Master 127.0.98.62:41515 requested a full tablet report, sending...
I20260927 00:39:05.606369 2822 ts_manager.cc:194] Registered new tserver with Master: 6908d321add442389cf4964e10937f35 (127.0.98.3:41507)
I20260927 00:39:05.606973 2822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60477
I20260927 00:39:05.610771 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:39:05.623942 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:05.622417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.0.98.62:41515)
0927 00:39:05.622542 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:05.622546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:05.622638 (+ 92us) server_negotiation.cc:407] Connection header received
0927 00:39:05.622750 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:05.622753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:05.622802 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:05.622887 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:05.623766 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:39:05.625002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2884
I20260927 00:39:05.630118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3018
I20260927 00:39:05.635071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3152
I20260927 00:39:05.639922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 2790
2026-09-27T00:39:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2086 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[3293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[3293](info): set up 2 sockets
Sep 27 00:39:05 dist-test-slave-xltq krb5kdc[3293](info): commencing operation
krb5kdc: starting...
W20260927 00:39:06.616053 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:06 dist-test-slave-xltq krb5kdc[3293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:06.741055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33005
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:06.857111 3306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:06.857398 3306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:06.857439 3306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:06.861670 3306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:06.861749 3306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:06.861768 3306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:06.861783 3306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:06.861799 3306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:06.867367 3306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:06.868512 3306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:06.869684 3306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:06.878906 3314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:06.878895 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:06.878894 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:06.878953 3306 server_base.cc:1034] running on GCE node
I20260927 00:39:06.879829 3306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:06.881070 3306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:06.882272 3306 hybrid_clock.cc:648] HybridClock initialized: now 1790469546882247 us; error 38 us; skew 500 ppm
Sep 27 00:39:06 dist-test-slave-xltq krb5kdc[3293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:06.885386 3306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:06.886386 3306 webserver.cc:460] Webserver started at http://127.0.98.62:38223/ using document root <none> and password file <none>
I20260927 00:39:06.886993 3306 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:06.887066 3306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:06.887276 3306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:06.889716 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "dc85dfc85b13418c84f25472bb255729"
format_stamp: "Formatted at 2026-09-27 00:39:06 on dist-test-slave-xltq"
server_key: "1d24557406afea6ca25e1c2ab90cfb34"
server_key_iv: "1e99c0fd03c69262d14c1c268d93ad53"
server_key_version: "encryptionkey@0"
I20260927 00:39:06.890475 3306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc85dfc85b13418c84f25472bb255729"
format_stamp: "Formatted at 2026-09-27 00:39:06 on dist-test-slave-xltq"
server_key: "1d24557406afea6ca25e1c2ab90cfb34"
server_key_iv: "1e99c0fd03c69262d14c1c268d93ad53"
server_key_version: "encryptionkey@0"
I20260927 00:39:06.895262 3306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:06.898066 3321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:06.899529 3306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:06.899658 3306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "dc85dfc85b13418c84f25472bb255729"
format_stamp: "Formatted at 2026-09-27 00:39:06 on dist-test-slave-xltq"
server_key: "1d24557406afea6ca25e1c2ab90cfb34"
server_key_iv: "1e99c0fd03c69262d14c1c268d93ad53"
server_key_version: "encryptionkey@0"
I20260927 00:39:06.899761 3306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:06.924463 3306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:06.927654 3306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:06.927817 3306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:06.934813 3306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33005
I20260927 00:39:06.935068 3373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33005 every 8 connection(s)
I20260927 00:39:06.935863 3306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:06.938297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3306
I20260927 00:39:06.938405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:06.938697 392 external_mini_cluster.cc:1428] Setting key 370e7f5e2c85c046887436009326d11e
I20260927 00:39:06.939052 3374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:06.945609 3374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729: Bootstrap starting.
Sep 27 00:39:06 dist-test-slave-xltq krb5kdc[3293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:06.948076 3374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:06.949008 3374 log.cc:824] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:06.951134 3374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729: No bootstrap required, opened a new log
I20260927 00:39:06.953262 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:06.940080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.62:33005)
0927 00:39:06.940664 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:06.940680 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:06.940719 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:39:06.941404 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:06.941423 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:06.941775 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:06.942101 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:06.942979 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:06.944024 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:06.944904 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:06.945385 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:06.945528 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:06.947996 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:06.948002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:06.948017 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:06.948050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:06.949902 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:06.950583 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:06.950589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:06.950595 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:06.950730 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:06.950996 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:06.950999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:06.951000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:06.951451 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:06.951662 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:06.951925 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:39:06.952436 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":159,"threads_started":1}
I20260927 00:39:06.955350 3374 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 00:39:06.955667 3374 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:06.955721 3374 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85dfc85b13418c84f25472bb255729, State: Initialized, Role: FOLLOWER
I20260927 00:39:06.956172 3374 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 00:39:06.956354 3374 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:06.956420 3374 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:06.956516 3374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:06.956749 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:39:06.957547 3374 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 00:39:06.957911 3374 leader_election.cc:302] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc85dfc85b13418c84f25472bb255729; no voters:
I20260927 00:39:06.958258 3374 leader_election.cc:288] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:06.958352 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:06.958600 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [term 1 LEADER]: Becoming Leader. State: Replica: dc85dfc85b13418c84f25472bb255729, State: Running, Role: LEADER
I20260927 00:39:06.959012 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } }
I20260927 00:39:06.959661 3374 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:06.960351 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc85dfc85b13418c84f25472bb255729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } } }
I20260927 00:39:06.960554 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:06.960764 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc85dfc85b13418c84f25472bb255729. Latest consensus state: current_term: 1 leader_uuid: "dc85dfc85b13418c84f25472bb255729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc85dfc85b13418c84f25472bb255729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33005 } } }
I20260927 00:39:06.960861 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:06.960975 3388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:06.964393 3388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:06.971103 3388 catalog_manager.cc:1347] Generated new cluster ID: aa44bb8b125b472399fcc5e51f176475
I20260927 00:39:06.971179 3388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:06.975148 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:06.969695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.98.62:33005)
0927 00:39:06.969805 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:06.969809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:06.969827 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:06.969921 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:06.969924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:06.969975 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:06.970073 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:06.970680 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:06.971511 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:06.972696 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:06.973044 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:06.973158 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:06.974042 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:06.974046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:06.974049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:06.974642 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:06.974681 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:06.974702 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:06.974795 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:06.974868 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:39:06.974938 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:06.991204 3388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:06.992170 3388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:07.004304 3388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc85dfc85b13418c84f25472bb255729: Generated new TSK 0
I20260927 00:39:07.005026 3388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:07.016947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3306
2026-09-27T00:39:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1375 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:07 dist-test-slave-xltq krb5kdc[3407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:07 dist-test-slave-xltq krb5kdc[3407](info): set up 2 sockets
Sep 27 00:39:07 dist-test-slave-xltq krb5kdc[3407](info): commencing operation
krb5kdc: starting...
W20260927 00:39:07.991988 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:08.115415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:08.230931 3420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:08.231297 3420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:08.231348 3420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:08.235635 3420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:08.235728 3420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:08.235750 3420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:08.235769 3420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:08.235786 3420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:08.241567 3420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:08.242774 3420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:08.243922 3420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:08.253249 3428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.253271 3425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.253268 3426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:08.253464 3420 server_base.cc:1034] running on GCE node
I20260927 00:39:08.254237 3420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:08.255506 3420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:08.256700 3420 hybrid_clock.cc:648] HybridClock initialized: now 1790469548256664 us; error 48 us; skew 500 ppm
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:08.259891 3420 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:08.261008 3420 webserver.cc:460] Webserver started at http://127.0.98.62:45281/ using document root <none> and password file <none>
I20260927 00:39:08.261621 3420 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:08.261694 3420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:08.261895 3420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:08.264040 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "8d679a7c3745418bbbeb9d46ee13b28a"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "073f31c903971b1f1747d0242abca26d"
server_key_iv: "623eb6881fdff45c0a26db614d1b59ee"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.264657 3420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d679a7c3745418bbbeb9d46ee13b28a"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "073f31c903971b1f1747d0242abca26d"
server_key_iv: "623eb6881fdff45c0a26db614d1b59ee"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.268787 3420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:08.271664 3435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.273129 3420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:08.273344 3420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "8d679a7c3745418bbbeb9d46ee13b28a"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "073f31c903971b1f1747d0242abca26d"
server_key_iv: "623eb6881fdff45c0a26db614d1b59ee"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.273466 3420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:08.284420 3420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:08.288327 3420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:08.288542 3420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:08.296383 3420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35325
I20260927 00:39:08.296663 3487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35325 every 8 connection(s)
I20260927 00:39:08.297502 3420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:08.300941 3488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:08.302181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3420
I20260927 00:39:08.302314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:08.302626 392 external_mini_cluster.cc:1428] Setting key 2d151be329bd31353d6dfa0e00968847
I20260927 00:39:08.307541 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a: Bootstrap starting.
I20260927 00:39:08.309942 3488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:08.310771 3488 log.cc:824] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:08.312990 3488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a: No bootstrap required, opened a new log
I20260927 00:39:08.316674 3488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } }
I20260927 00:39:08.316949 3488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:08.317003 3488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d679a7c3745418bbbeb9d46ee13b28a, State: Initialized, Role: FOLLOWER
I20260927 00:39:08.317452 3488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } }
I20260927 00:39:08.317580 3488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:08.317641 3488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:08.317718 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:08.304320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.98.62:35325)
0927 00:39:08.304878 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:08.304895 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:08.304937 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:39:08.305645 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:08.305666 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:08.306039 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:08.306416 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:08.307339 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.308475 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.309402 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.309916 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.310083 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:08.312992 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:08.312998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:08.313013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:08.313044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:08.314682 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.315162 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.315167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.315172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.315305 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.315565 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.315569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.315570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.315985 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:08.316189 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:08.316520 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 00:39:08.316826 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":164,"threads_started":1}
I20260927 00:39:08.317734 3488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:08.318859 3488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } }
I20260927 00:39:08.319235 3488 leader_election.cc:302] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d679a7c3745418bbbeb9d46ee13b28a; no voters:
I20260927 00:39:08.319584 3488 leader_election.cc:288] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:08.319695 3493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:08.319988 3493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [term 1 LEADER]: Becoming Leader. State: Replica: 8d679a7c3745418bbbeb9d46ee13b28a, State: Running, Role: LEADER
I20260927 00:39:08.320399 3493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } }
I20260927 00:39:08.320855 3488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:08.322065 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d679a7c3745418bbbeb9d46ee13b28a. Latest consensus state: current_term: 1 leader_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } } }
I20260927 00:39:08.322198 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:08.322492 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d679a7c3745418bbbeb9d46ee13b28a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35325 } } }
I20260927 00:39:08.322597 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:08.322669 3502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:08.325949 3502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:08.332088 3502 catalog_manager.cc:1347] Generated new cluster ID: 2d8deafd38384b129efbd4340f560a4c
I20260927 00:39:08.332165 3502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:08.343850 3502 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:08.344945 3502 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:08.354316 3502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d679a7c3745418bbbeb9d46ee13b28a: Generated new TSK 0
I20260927 00:39:08.355221 3502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:08.396348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:08.511006 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:08.511294 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:08.511337 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:08.515483 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:08.515566 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:08.515627 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:08.521062 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:08.522249 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:08.523294 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:08.533003 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.533265 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.533012 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:08.533272 3514 server_base.cc:1034] running on GCE node
I20260927 00:39:08.533871 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:08.534551 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:08.535719 3514 hybrid_clock.cc:648] HybridClock initialized: now 1790469548535683 us; error 61 us; skew 500 ppm
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:08.538723 3514 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:08.539713 3514 webserver.cc:460] Webserver started at http://127.0.98.1:46257/ using document root <none> and password file <none>
I20260927 00:39:08.540364 3514 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:08.540444 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:08.540654 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:08.542856 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "5992029f3b474ef285c628068406a79c"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "76a52e8f8b63a16bed446fb6c8a7a764"
server_key_iv: "35d50b16c38ce72970ce9c058ef44247"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.543403 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5992029f3b474ef285c628068406a79c"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "76a52e8f8b63a16bed446fb6c8a7a764"
server_key_iv: "35d50b16c38ce72970ce9c058ef44247"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.547307 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:39:08.550055 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.551462 3514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:39:08.551596 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "5992029f3b474ef285c628068406a79c"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "76a52e8f8b63a16bed446fb6c8a7a764"
server_key_iv: "35d50b16c38ce72970ce9c058ef44247"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.551708 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:08.565912 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:08.569357 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:08.569556 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:08.570178 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:08.571053 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:08.571100 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.571156 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:08.571183 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.580452 3514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40069
I20260927 00:39:08.580475 3642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40069 every 8 connection(s)
I20260927 00:39:08.581496 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:08.582778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3514
I20260927 00:39:08.582899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:08.583196 392 external_mini_cluster.cc:1428] Setting key 5c8f04a5a1498b41c76e459ce28d8d4e
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:08.601615 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:08.583632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38755 (local address 127.0.98.62:35325)
0927 00:39:08.583794 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:08.583798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:08.584944 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 00:39:08.586352 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:08.586355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:08.586413 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:08.586486 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:08.588044 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.588698 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.590574 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.590845 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.590933 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:08.597433 (+ 6500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:08.597437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:08.597439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:08.597471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:08.599008 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.599739 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.599743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.599744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.599843 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.600271 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.600275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.600277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.600560 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:08.600672 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:08.601288 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:39:08.601435 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:39:08.602530 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:08.584172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35325 (local address 127.0.98.1:38755)
0927 00:39:08.584794 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:08.584842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:08.586123 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:08.586685 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:08.586693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:08.587148 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:08.587860 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:08.587881 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.588851 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.588855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:08.590405 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:08.590421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.591025 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.591030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:08.591269 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:08.591941 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:08.591966 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:08.597124 (+ 5158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:08.599176 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:08.599182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:08.599199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:08.599602 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:08.599941 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:08.599944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:08.599947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:08.600118 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:08.600699 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:08.600705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:08.601007 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 00:39:08.601198 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 00:39:08.601597 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260927 00:39:08.604010 3643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35325
I20260927 00:39:08.604313 3643 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:08.604876 3643 heartbeater.cc:511] Master 127.0.98.62:35325 requested a full tablet report, sending...
I20260927 00:39:08.606597 3452 ts_manager.cc:194] Registered new tserver with Master: 5992029f3b474ef285c628068406a79c (127.0.98.1:40069)
I20260927 00:39:08.607780 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:08.615727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:08.727337 3648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:08.727656 3648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:08.727716 3648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:08.732146 3648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:08.732265 3648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:08.732369 3648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:08.738168 3648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:08.739337 3648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:08.740477 3648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:08.750676 3656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.750890 3653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:08.750815 3654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:08.750999 3648 server_base.cc:1034] running on GCE node
I20260927 00:39:08.751642 3648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:08.752457 3648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:08.753664 3648 hybrid_clock.cc:648] HybridClock initialized: now 1790469548753628 us; error 49 us; skew 500 ppm
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:08.762841 3648 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:08.763813 3648 webserver.cc:460] Webserver started at http://127.0.98.2:36403/ using document root <none> and password file <none>
I20260927 00:39:08.764434 3648 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:08.764503 3648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:08.764674 3648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:08.766754 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c99863f8c5444128483ae5ad2a80896"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "e121b16306da628b35b3348fb3f0a726"
server_key_iv: "d91c7853a0d18f58aab66d6ed893f23a"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.767295 3648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c99863f8c5444128483ae5ad2a80896"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "e121b16306da628b35b3348fb3f0a726"
server_key_iv: "d91c7853a0d18f58aab66d6ed893f23a"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.771276 3648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:39:08.774044 3663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.775501 3648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:08.775619 3648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "7c99863f8c5444128483ae5ad2a80896"
format_stamp: "Formatted at 2026-09-27 00:39:08 on dist-test-slave-xltq"
server_key: "e121b16306da628b35b3348fb3f0a726"
server_key_iv: "d91c7853a0d18f58aab66d6ed893f23a"
server_key_version: "encryptionkey@0"
I20260927 00:39:08.775723 3648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:08.810467 3648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:08.813910 3648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:08.814077 3648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:08.814661 3648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:08.815500 3648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:08.815546 3648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.815587 3648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:08.815600 3648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:08.824682 3648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43415
I20260927 00:39:08.824726 3776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43415 every 8 connection(s)
I20260927 00:39:08.825598 3648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:08.832952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3648
I20260927 00:39:08.833062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:08.833318 392 external_mini_cluster.cc:1428] Setting key cb0b9b492cf048a11f991ea599da8d0c
Sep 27 00:39:08 dist-test-slave-xltq krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:08.841683 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:08.827478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50847 (local address 127.0.98.62:35325)
0927 00:39:08.827622 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:08.827627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:08.828683 (+ 1056us) server_negotiation.cc:407] Connection header received
0927 00:39:08.830132 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:08.830135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:08.830188 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:08.830266 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:08.831502 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.832093 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.833504 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.833813 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.833929 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:08.837031 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:08.837035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:08.837038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:08.837071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:08.838687 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.839500 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.839503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.839504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.839600 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:08.840083 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:08.840086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:08.840087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:08.840450 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:08.840538 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:08.841433 (+ 895us) server_negotiation.cc:299] Negotiation successful
0927 00:39:08.841554 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:39:08.842953 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:08.827977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35325 (local address 127.0.98.2:50847)
0927 00:39:08.828515 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:08.828569 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:08.829905 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:08.830421 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:08.830428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:08.830810 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:08.831338 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:08.831353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.832295 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.832299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:08.833371 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:08.833383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:08.833994 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:08.833999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:08.834214 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:08.834897 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:08.834914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:08.836812 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:08.838858 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:08.838865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:08.838885 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:08.839343 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:08.839702 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:08.839705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:08.839707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:08.839953 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:08.840566 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:08.840575 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:08.840993 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 00:39:08.841227 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 00:39:08.841654 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:08.844431 3777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35325
I20260927 00:39:08.844682 3777 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:08.845206 3777 heartbeater.cc:511] Master 127.0.98.62:35325 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:08.846331 3452 ts_manager.cc:194] Registered new tserver with Master: 7c99863f8c5444128483ae5ad2a80896 (127.0.98.2:43415)
I20260927 00:39:08.846927 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:08.863821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:08.977607 3782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:08.977892 3782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:08.977939 3782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:08.982642 3782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:08.982717 3782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:08.982784 3782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:08.988462 3782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:08.989997 3782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:08.991166 3782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:09.001173 3790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:09.001192 3787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:09.001279 3782 server_base.cc:1034] running on GCE node
W20260927 00:39:09.001192 3788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:09.001824 3782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:09.002473 3782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:09.003638 3782 hybrid_clock.cc:648] HybridClock initialized: now 1790469549003613 us; error 36 us; skew 500 ppm
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:09.006543 3782 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:09.007472 3782 webserver.cc:460] Webserver started at http://127.0.98.3:35767/ using document root <none> and password file <none>
I20260927 00:39:09.008023 3782 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:09.008092 3782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:09.008334 3782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:09.010421 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "fec40b2470a843139757156debd295c4"
format_stamp: "Formatted at 2026-09-27 00:39:09 on dist-test-slave-xltq"
server_key: "a35f7703e0183b4cc2c51b3de02ab588"
server_key_iv: "24140fd36c45a39a76a884f0801476cc"
server_key_version: "encryptionkey@0"
I20260927 00:39:09.010958 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fec40b2470a843139757156debd295c4"
format_stamp: "Formatted at 2026-09-27 00:39:09 on dist-test-slave-xltq"
server_key: "a35f7703e0183b4cc2c51b3de02ab588"
server_key_iv: "24140fd36c45a39a76a884f0801476cc"
server_key_version: "encryptionkey@0"
I20260927 00:39:09.014812 3782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 00:39:09.017309 3797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:09.018615 3782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 00:39:09.018723 3782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "fec40b2470a843139757156debd295c4"
format_stamp: "Formatted at 2026-09-27 00:39:09 on dist-test-slave-xltq"
server_key: "a35f7703e0183b4cc2c51b3de02ab588"
server_key_iv: "24140fd36c45a39a76a884f0801476cc"
server_key_version: "encryptionkey@0"
I20260927 00:39:09.018823 3782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:09.032362 3782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:09.035480 3782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:09.035663 3782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:09.036290 3782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:09.037132 3782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:09.037184 3782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:09.037237 3782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:09.037263 3782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:09.046281 3782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40199
I20260927 00:39:09.046312 3910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40199 every 8 connection(s)
I20260927 00:39:09.047215 3782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:09.050316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3782
I20260927 00:39:09.050525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:09.050864 392 external_mini_cluster.cc:1428] Setting key 89755d29ca321166e8ef3117ca009fa2
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:09.061981 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:09.049249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53483 (local address 127.0.98.62:35325)
0927 00:39:09.049427 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:09.049443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:09.050514 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 00:39:09.052370 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:09.052373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:09.052439 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:09.052512 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:09.053856 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.054366 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.055566 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.055834 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.055920 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:09.058392 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:09.058396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:09.058398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:09.058440 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:09.059876 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:09.060415 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:09.060419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:09.060420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:09.060501 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:09.060829 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:09.060833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:09.060835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:09.061038 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:09.061123 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:09.061664 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 00:39:09.061804 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:39:09.062836 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:09.049715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35325 (local address 127.0.98.3:53483)
0927 00:39:09.050306 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:09.050387 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:09.052109 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:09.052714 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:09.052722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:09.053203 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:09.053714 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:09.053729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.054496 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.054500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:09.055450 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:09.055460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.055994 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.055999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:09.056172 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:09.056805 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:09.056822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:09.058228 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:09.059988 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:09.059995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:09.060013 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:09.060317 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:09.060583 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:09.060586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:09.060588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:09.060735 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:09.061136 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:09.061143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:09.061415 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 00:39:09.061590 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 00:39:09.061991 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":141,"threads_started":1}
I20260927 00:39:09.064116 3911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35325
I20260927 00:39:09.064468 3911 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:09.065048 3911 heartbeater.cc:511] Master 127.0.98.62:35325 requested a full tablet report, sending...
I20260927 00:39:09.066010 3451 ts_manager.cc:194] Registered new tserver with Master: fec40b2470a843139757156debd295c4 (127.0.98.3:40199)
I20260927 00:39:09.066582 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53483
I20260927 00:39:09.077169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469549, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469549, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:09.102396 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:09.093708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.62:35325)
0927 00:39:09.093826 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:09.093829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:09.093931 (+ 102us) server_negotiation.cc:407] Connection header received
0927 00:39:09.094101 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:09.094105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:09.094155 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:09.094239 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:09.094886 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.095485 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.096499 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:09.096722 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:09.096817 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:09.099211 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:09.099214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:09.099216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:09.099243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:09.100677 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:09.101155 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:09.101159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:09.101160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:09.101245 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:09.101500 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:09.101502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:09.101503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:09.101745 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:09.101851 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:09.102096 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 00:39:09.102200 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:39:09.102789 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56210
I20260927 00:39:09.103758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3514
I20260927 00:39:09.108520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3648
I20260927 00:39:09.113297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3782
I20260927 00:39:09.117908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3420
2026-09-27T00:39:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2102 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3923](info): set up 2 sockets
Sep 27 00:39:09 dist-test-slave-xltq krb5kdc[3923](info): commencing operation
krb5kdc: starting...
W20260927 00:39:10.076099 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:10.224185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:10.337415 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:10.337739 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:10.337800 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:10.342154 3936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:10.342238 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:10.342267 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:10.342288 3936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:10.342307 3936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:10.347966 3936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:10.349176 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:10.350266 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:10.358945 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:10.359059 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:10.359127 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:10.359203 3936 server_base.cc:1034] running on GCE node
I20260927 00:39:10.359958 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:10.361169 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:10.362334 3936 hybrid_clock.cc:648] HybridClock initialized: now 1790469550362318 us; error 35 us; skew 500 ppm
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:10.365197 3936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:10.366144 3936 webserver.cc:460] Webserver started at http://127.0.98.62:35707/ using document root <none> and password file <none>
I20260927 00:39:10.366690 3936 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:10.366757 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:10.366967 3936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:10.369212 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "ce497a9c97fc4993901a023de072f020"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "f5051c4c4a7447a7de00156096b9257b"
server_key_iv: "b68c77b61571748fce8236e22a476989"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.369767 3936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce497a9c97fc4993901a023de072f020"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "f5051c4c4a7447a7de00156096b9257b"
server_key_iv: "b68c77b61571748fce8236e22a476989"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.373643 3936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:10.376156 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.377635 3936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:10.377758 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ce497a9c97fc4993901a023de072f020"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "f5051c4c4a7447a7de00156096b9257b"
server_key_iv: "b68c77b61571748fce8236e22a476989"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.377866 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:10.395812 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:10.399058 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:10.399248 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:10.406344 3936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45391
I20260927 00:39:10.406401 4003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45391 every 8 connection(s)
I20260927 00:39:10.407351 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:10.410323 4004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:10.410846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 3936
I20260927 00:39:10.410965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:10.411260 392 external_mini_cluster.cc:1428] Setting key df2f3666605e6d8df42a3f4abc930f51
I20260927 00:39:10.416739 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020: Bootstrap starting.
I20260927 00:39:10.419420 4004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:10.420313 4004 log.cc:824] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:10.422267 4004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020: No bootstrap required, opened a new log
I20260927 00:39:10.425314 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:10.412801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.98.62:45391)
0927 00:39:10.413251 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:10.413267 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:10.413304 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:39:10.413964 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:10.413983 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:10.414326 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:10.414627 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:10.415483 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.416547 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.417685 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.418170 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.418282 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:10.420985 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:10.420991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:10.421006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:10.421033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:10.422667 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.423074 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.423079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.423084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.423193 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.423435 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.423438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.423440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.423825 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:10.424007 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:10.424308 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:39:10.424586 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":126,"threads_started":1}
I20260927 00:39:10.425843 4004 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260927 00:39:10.426096 4004 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:10.426196 4004 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce497a9c97fc4993901a023de072f020, State: Initialized, Role: FOLLOWER
I20260927 00:39:10.426666 4004 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260927 00:39:10.426797 4004 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:10.426862 4004 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:10.426966 4004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:10.427944 4004 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260927 00:39:10.428299 4004 leader_election.cc:302] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce497a9c97fc4993901a023de072f020; no voters:
I20260927 00:39:10.428586 4004 leader_election.cc:288] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:10.428682 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:10.428875 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [term 1 LEADER]: Becoming Leader. State: Replica: ce497a9c97fc4993901a023de072f020, State: Running, Role: LEADER
I20260927 00:39:10.429214 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260927 00:39:10.429754 4004 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:10.430792 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce497a9c97fc4993901a023de072f020. Latest consensus state: current_term: 1 leader_uuid: "ce497a9c97fc4993901a023de072f020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } } }
I20260927 00:39:10.430917 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:10.431272 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:10.431241 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce497a9c97fc4993901a023de072f020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce497a9c97fc4993901a023de072f020" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } } }
I20260927 00:39:10.431372 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:10.435201 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:10.440677 4018 catalog_manager.cc:1347] Generated new cluster ID: 21b6349c3c7f4abb8220c4167e70f2b2
I20260927 00:39:10.440763 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:10.449457 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:10.450610 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:10.454370 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce497a9c97fc4993901a023de072f020: Generated new TSK 0
I20260927 00:39:10.455068 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:10.484855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:10.596374 4030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:10.596664 4030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:10.596709 4030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:10.600867 4030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:10.600934 4030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:10.600996 4030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:10.606416 4030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:10.607504 4030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:10.608536 4030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:10.618451 4035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:10.618498 4038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:10.618582 4030 server_base.cc:1034] running on GCE node
W20260927 00:39:10.618748 4036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:10.619181 4030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:10.619868 4030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:10.621027 4030 hybrid_clock.cc:648] HybridClock initialized: now 1790469550620997 us; error 45 us; skew 500 ppm
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:10.623662 4030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:10.624629 4030 webserver.cc:460] Webserver started at http://127.0.98.1:43185/ using document root <none> and password file <none>
I20260927 00:39:10.625169 4030 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:10.625232 4030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:10.625414 4030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:10.627476 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba37f0aa375743cc972c0aaa021714bc"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "1b1242af9c23e0ebf0c5505b63ab63fb"
server_key_iv: "f180957330a7e59ca56b2a0884ea715b"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.628017 4030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba37f0aa375743cc972c0aaa021714bc"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "1b1242af9c23e0ebf0c5505b63ab63fb"
server_key_iv: "f180957330a7e59ca56b2a0884ea715b"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.631912 4030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:10.634460 4045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.635809 4030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:10.635958 4030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "ba37f0aa375743cc972c0aaa021714bc"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "1b1242af9c23e0ebf0c5505b63ab63fb"
server_key_iv: "f180957330a7e59ca56b2a0884ea715b"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.636096 4030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:10.648473 4030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:10.651916 4030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:10.652139 4030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:10.652762 4030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:10.653617 4030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:10.653666 4030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.653721 4030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:10.653749 4030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.662436 4030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46039
I20260927 00:39:10.662463 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46039 every 8 connection(s)
I20260927 00:39:10.663370 4030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:10.671268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4030
I20260927 00:39:10.671392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:10.671703 392 external_mini_cluster.cc:1428] Setting key 31386885b609cac1daef7a71498149d1
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:10.677881 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:10.665303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44479 (local address 127.0.98.62:45391)
0927 00:39:10.665457 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:10.665460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:10.666415 (+ 955us) server_negotiation.cc:407] Connection header received
0927 00:39:10.667826 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:10.667829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:10.667886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:10.667964 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:10.669250 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.669797 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.670966 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.671251 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.671352 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:10.674213 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:10.674216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:10.674219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:10.674243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:10.675639 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.676247 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.676251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.676253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.676331 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.676691 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.676694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.676695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.676876 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:10.676949 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:10.677640 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 00:39:10.677758 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:39:10.679139 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:10.665732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.1:44479)
0927 00:39:10.666248 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:10.666294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:10.667602 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:10.668162 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:10.668170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:10.668600 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:10.669107 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:10.669122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.669925 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.669928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:10.670844 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:10.670854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.671452 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.671457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:10.671771 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:10.672450 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:10.672467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:10.674036 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:10.675764 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:10.675770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:10.675787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:10.676122 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:10.676409 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:10.676412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:10.676414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:10.676594 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:10.676972 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:10.676979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:10.677351 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 00:39:10.677600 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 00:39:10.677987 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":125,"threads_started":1}
I20260927 00:39:10.680481 4159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
I20260927 00:39:10.680760 4159 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:10.681293 4159 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:10.682920 3968 ts_manager.cc:194] Registered new tserver with Master: ba37f0aa375743cc972c0aaa021714bc (127.0.98.1:46039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:10.684295 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:10.698866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:10.811565 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:10.811856 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:10.811918 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:10.816145 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:10.816246 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:10.816339 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:10.821882 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:10.823033 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:10.824097 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:10.834105 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:10.834107 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:10.834107 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:10.834408 4164 server_base.cc:1034] running on GCE node
I20260927 00:39:10.834978 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:10.835652 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:10.836826 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790469550836802 us; error 36 us; skew 500 ppm
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:10.839982 4164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:10.841150 4164 webserver.cc:460] Webserver started at http://127.0.98.2:41809/ using document root <none> and password file <none>
I20260927 00:39:10.841750 4164 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:10.841811 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:10.841986 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:10.844334 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "c906f351ec0f4997bce1a19fc8d883c9"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "a02374a7c1c3d013aed849533c2682da"
server_key_iv: "3d3a3b5c154d06a586edb5cc991927ca"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.844957 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c906f351ec0f4997bce1a19fc8d883c9"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "a02374a7c1c3d013aed849533c2682da"
server_key_iv: "3d3a3b5c154d06a586edb5cc991927ca"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.849313 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 00:39:10.851992 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.853364 4164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:39:10.853473 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "c906f351ec0f4997bce1a19fc8d883c9"
format_stamp: "Formatted at 2026-09-27 00:39:10 on dist-test-slave-xltq"
server_key: "a02374a7c1c3d013aed849533c2682da"
server_key_iv: "3d3a3b5c154d06a586edb5cc991927ca"
server_key_version: "encryptionkey@0"
I20260927 00:39:10.853574 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:10.876250 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:10.879573 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:10.879745 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:10.880383 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:10.881239 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:10.881286 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.881330 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:10.881352 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:10.889993 4164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40333
I20260927 00:39:10.890033 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40333 every 8 connection(s)
I20260927 00:39:10.890945 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:10.895563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4164
I20260927 00:39:10.895685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:10.895942 392 external_mini_cluster.cc:1428] Setting key 8a095e8debe9fa3984f26379160ca8f0
Sep 27 00:39:10 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:10.905921 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:10.892836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48919 (local address 127.0.98.62:45391)
0927 00:39:10.892956 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:10.892959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:10.893900 (+ 941us) server_negotiation.cc:407] Connection header received
0927 00:39:10.895400 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:10.895404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:10.895463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:10.895543 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:10.896989 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.897591 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.898830 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.899059 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.899136 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:10.902123 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:10.902126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:10.902128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:10.902185 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:10.903591 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.904244 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.904248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.904250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.904329 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:10.904725 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:10.904728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:10.904729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:10.904919 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:10.904996 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:10.905665 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 00:39:10.905798 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:39:10.906828 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:10.893259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.2:48919)
0927 00:39:10.893739 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:10.893786 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:10.895135 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:10.895766 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:10.895775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:10.896186 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:10.896850 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:10.896866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.897718 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.897721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:10.898687 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:10.898699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:10.899201 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:10.899205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:10.899414 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:10.900070 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:10.900091 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:10.901938 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:10.903717 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:10.903724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:10.903743 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:10.904100 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:10.904420 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:10.904424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:10.904426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:10.904622 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:10.905018 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:10.905025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:10.905395 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 00:39:10.905579 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:39:10.905862 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:10.908190 4293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
I20260927 00:39:10.908474 4293 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:10.909147 4293 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
I20260927 00:39:10.910276 3968 ts_manager.cc:194] Registered new tserver with Master: c906f351ec0f4997bce1a19fc8d883c9 (127.0.98.2:40333)
I20260927 00:39:10.911029 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:10.923084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:11.037667 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:11.037976 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:11.038023 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:11.042390 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:11.042472 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:11.042536 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:11.048308 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:11.049428 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:11.050424 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:11.061026 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:11.061026 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:11.061273 4298 server_base.cc:1034] running on GCE node
W20260927 00:39:11.061300 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:11.061772 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:11.062558 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:11.063767 4298 hybrid_clock.cc:648] HybridClock initialized: now 1790469551063734 us; error 42 us; skew 500 ppm
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:11.067049 4298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:11.068111 4298 webserver.cc:460] Webserver started at http://127.0.98.3:35955/ using document root <none> and password file <none>
I20260927 00:39:11.068753 4298 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:11.068830 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:11.069036 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:11.071254 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "33137389cf7f4aa2b3e34785227aaa86"
format_stamp: "Formatted at 2026-09-27 00:39:11 on dist-test-slave-xltq"
server_key: "1d2e0fbe80258b2e8d15fa1cd008ca0d"
server_key_iv: "8b17046997723abf259605e401bdbbd1"
server_key_version: "encryptionkey@0"
I20260927 00:39:11.071854 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33137389cf7f4aa2b3e34785227aaa86"
format_stamp: "Formatted at 2026-09-27 00:39:11 on dist-test-slave-xltq"
server_key: "1d2e0fbe80258b2e8d15fa1cd008ca0d"
server_key_iv: "8b17046997723abf259605e401bdbbd1"
server_key_version: "encryptionkey@0"
I20260927 00:39:11.076489 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 00:39:11.079227 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:11.080835 4298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:11.080960 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "33137389cf7f4aa2b3e34785227aaa86"
format_stamp: "Formatted at 2026-09-27 00:39:11 on dist-test-slave-xltq"
server_key: "1d2e0fbe80258b2e8d15fa1cd008ca0d"
server_key_iv: "8b17046997723abf259605e401bdbbd1"
server_key_version: "encryptionkey@0"
I20260927 00:39:11.081092 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:11.096588 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:11.099640 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:11.099833 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:11.100492 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:11.101356 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:11.101404 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:11.101459 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:11.101487 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:11.110096 4298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43187
I20260927 00:39:11.110180 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43187 every 8 connection(s)
I20260927 00:39:11.111044 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:11.119699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4298
I20260927 00:39:11.119831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:11.120092 392 external_mini_cluster.cc:1428] Setting key 37042594aa0fa104a73fd036fa22e027
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:11.125475 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:11.112995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37241 (local address 127.0.98.62:45391)
0927 00:39:11.113150 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:11.113154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:11.114046 (+ 892us) server_negotiation.cc:407] Connection header received
0927 00:39:11.115461 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:11.115464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:11.115515 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:11.115595 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:11.116910 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.117454 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.118692 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.118932 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.119026 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:11.121855 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:11.121858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:11.121860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:11.121886 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:11.123380 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.123903 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.123906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.123908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.123983 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.124360 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.124364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.124365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.124550 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:11.124635 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:11.125172 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 00:39:11.125310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:39:11.126325 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:11.113403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.3:37241)
0927 00:39:11.113884 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:11.113929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:11.115251 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:11.115803 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:11.115810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:11.116187 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:11.116747 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:11.116761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.117571 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.117574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:11.118560 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:11.118569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.119104 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.119108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:11.119271 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:11.119991 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:11.120008 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:11.121686 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:11.123497 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:11.123504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:11.123522 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:11.123811 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:11.124064 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:11.124067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:11.124069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:11.124264 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:11.124648 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:11.124656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:11.124933 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 00:39:11.125102 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:39:11.125457 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260927 00:39:11.127521 4427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
I20260927 00:39:11.127791 4427 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:11.128389 4427 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
I20260927 00:39:11.129503 3968 ts_manager.cc:194] Registered new tserver with Master: 33137389cf7f4aa2b3e34785227aaa86 (127.0.98.3:43187)
I20260927 00:39:11.130148 3968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37241
I20260927 00:39:11.134241 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[3923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:39:11.161056 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:11.150643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.98.1:46039)
0927 00:39:11.150961 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:11.150973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:11.150996 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:39:11.151053 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:11.151058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:11.151263 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:11.151392 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:11.152291 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.153203 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.154225 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.154656 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.154763 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:11.157433 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:11.157444 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:11.157458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:11.157488 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:11.159087 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.159562 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.159568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.159570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.159718 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.159977 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.159979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.159980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.160312 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:11.160553 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:11.160720 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 00:39:11.160879 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[3923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:11.172470 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:11.164004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.0.98.62:45391)
0927 00:39:11.164177 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:11.164181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:11.164195 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:11.164276 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:11.164278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:11.164325 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:11.164409 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:11.165049 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.165576 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.166412 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:11.166698 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:11.166787 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:11.169241 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:11.169245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:11.169247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:11.169272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:11.170744 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.171229 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.171232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.171233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.171315 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:11.171606 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:11.171609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:11.171610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:11.171872 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:11.171973 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:11.172179 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:39:11.172347 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:39:11.172883 3968 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45840
I20260927 00:39:11.173527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4030
I20260927 00:39:11.178822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4164
I20260927 00:39:11.183378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4298
I20260927 00:39:11.188086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 3936
2026-09-27T00:39:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2070 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[4439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[4439](info): set up 2 sockets
Sep 27 00:39:11 dist-test-slave-xltq krb5kdc[4439](info): commencing operation
krb5kdc: starting...
W20260927 00:39:12.160113 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:12.327723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:12.444475 4452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:12.444790 4452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:12.444865 4452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:12.449260 4452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:12.449363 4452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:12.449393 4452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:12.449412 4452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:12.449433 4452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:12.455359 4452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:12.456578 4452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:12.457662 4452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:12.466442 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:12.466485 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:12.466565 4452 server_base.cc:1034] running on GCE node
W20260927 00:39:12.466485 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:12.467309 4452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:12.468680 4452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:12.469883 4452 hybrid_clock.cc:648] HybridClock initialized: now 1790469552469865 us; error 34 us; skew 500 ppm
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:12.472898 4452 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:12.473685 4452 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:12.473759 4452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:12.473968 4452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:12.476302 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "859b63d2d6cb41119979d5084e826861"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "73fe9430e3ab3e088156fd4419cdad39"
server_key_iv: "d9d41beb7946ee2989f85cbf97c31dac"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.476933 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "859b63d2d6cb41119979d5084e826861"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "73fe9430e3ab3e088156fd4419cdad39"
server_key_iv: "d9d41beb7946ee2989f85cbf97c31dac"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.480851 4452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 00:39:12.483515 4466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:12.484997 4452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:12.485127 4452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "859b63d2d6cb41119979d5084e826861"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "73fe9430e3ab3e088156fd4419cdad39"
server_key_iv: "d9d41beb7946ee2989f85cbf97c31dac"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.485252 4452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:12.499929 4452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:12.503449 4452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:12.503631 4452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:12.510589 4452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34419
I20260927 00:39:12.510841 4518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34419 every 8 connection(s)
I20260927 00:39:12.511278 4452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:12.514146 4519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:12.514566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4452
I20260927 00:39:12.514662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:12.514930 392 external_mini_cluster.cc:1428] Setting key 59d4be1ac9811422ab7cd76e33e78713
I20260927 00:39:12.521026 4519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: Bootstrap starting.
I20260927 00:39:12.523538 4519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: Neither blocks nor log segments found. Creating new log.
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:12.524538 4519 log.cc:824] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:12.526731 4519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: No bootstrap required, opened a new log
I20260927 00:39:12.529392 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:12.516328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.0.98.62:34419)
0927 00:39:12.516855 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:12.516873 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:12.516916 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:39:12.517618 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:12.517638 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:12.518007 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:12.518343 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:12.519367 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.520458 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.521402 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.521914 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.522064 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:12.524865 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:12.524871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:12.524887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:12.524917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:12.526550 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:12.527040 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:12.527045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:12.527050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:12.527159 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:12.527424 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:12.527426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:12.527427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:12.527822 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:12.527995 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:12.528294 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 00:39:12.528625 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":137,"threads_started":1}
I20260927 00:39:12.530735 4519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260927 00:39:12.530979 4519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:12.531035 4519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859b63d2d6cb41119979d5084e826861, State: Initialized, Role: FOLLOWER
I20260927 00:39:12.531451 4519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260927 00:39:12.531584 4519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:12.531648 4519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:12.531744 4519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:12.532717 4519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260927 00:39:12.533053 4519 leader_election.cc:302] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 859b63d2d6cb41119979d5084e826861; no voters:
I20260927 00:39:12.533354 4519 leader_election.cc:288] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:12.533463 4524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:12.533650 4524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [term 1 LEADER]: Becoming Leader. State: Replica: 859b63d2d6cb41119979d5084e826861, State: Running, Role: LEADER
I20260927 00:39:12.533965 4524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } }
I20260927 00:39:12.534533 4519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:12.535573 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 859b63d2d6cb41119979d5084e826861. Latest consensus state: current_term: 1 leader_uuid: "859b63d2d6cb41119979d5084e826861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } } }
I20260927 00:39:12.535704 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:12.535948 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "859b63d2d6cb41119979d5084e826861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859b63d2d6cb41119979d5084e826861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34419 } } }
I20260927 00:39:12.536028 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:12.536357 4533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:12.539242 4533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:12.544860 4533 catalog_manager.cc:1347] Generated new cluster ID: e332cd90f0834eea966026e2c9013474
I20260927 00:39:12.544951 4533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:12.571900 4533 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:12.573403 4533 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:12.578984 4533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: Generated new TSK 0
I20260927 00:39:12.579926 4533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:12.615451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:12.727720 4545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:12.728029 4545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:12.728076 4545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:12.732375 4545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:12.732446 4545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:12.732514 4545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:12.738288 4545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:12.739416 4545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:12.740480 4545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:12.750113 4551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:12.750113 4550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:12.750133 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:12.750396 4545 server_base.cc:1034] running on GCE node
I20260927 00:39:12.750965 4545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:12.751657 4545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:12.752854 4545 hybrid_clock.cc:648] HybridClock initialized: now 1790469552752829 us; error 35 us; skew 500 ppm
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:12.756003 4545 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:12.756872 4545 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:12.756958 4545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:12.757175 4545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:12.759625 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4e319c9471d41a3b74acfdbde6d24e3"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "3ad14ceab791f8b197bc15ae1b67e560"
server_key_iv: "7b15a45f9ca853fecc10fc5b32c99d74"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.760237 4545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4e319c9471d41a3b74acfdbde6d24e3"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "3ad14ceab791f8b197bc15ae1b67e560"
server_key_iv: "7b15a45f9ca853fecc10fc5b32c99d74"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.764318 4545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:39:12.767223 4559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:12.768921 4545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:12.769053 4545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "b4e319c9471d41a3b74acfdbde6d24e3"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "3ad14ceab791f8b197bc15ae1b67e560"
server_key_iv: "7b15a45f9ca853fecc10fc5b32c99d74"
server_key_version: "encryptionkey@0"
I20260927 00:39:12.769181 4545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:12.800112 4545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:12.803975 4545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:12.804160 4545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:12.804806 4545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:12.805686 4545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:12.805732 4545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:12.805773 4545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:12.805792 4545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:12.815755 4545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35305
I20260927 00:39:12.815773 4672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35305 every 8 connection(s)
I20260927 00:39:12.816565 4545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:12.822167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4545
I20260927 00:39:12.822304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:12.822603 392 external_mini_cluster.cc:1428] Setting key 10fb66c09dbbd29bbd963f84314dcf4a
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:12.832988 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:12.818520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57241 (local address 127.0.98.62:34419)
0927 00:39:12.818684 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:12.818688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:12.819713 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 00:39:12.821186 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:12.821190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:12.821252 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:12.821333 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:12.822920 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.823548 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.825284 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.825532 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.825657 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:12.828747 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:12.828752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:12.828755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:12.828790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:12.830491 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:12.831114 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:12.831118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:12.831120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:12.831214 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:12.831605 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:12.831608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:12.831610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:12.831874 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:12.831970 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:12.832728 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 00:39:12.832854 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:39:12.833917 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:12.818902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34419 (local address 127.0.98.1:57241)
0927 00:39:12.819520 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:12.819581 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:12.820911 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:12.821550 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:12.821557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:12.821990 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:12.822746 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:12.822769 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.823680 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.823683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:12.825088 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:12.825102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:12.825722 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:12.825727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:12.825917 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:12.826605 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:12.826623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:12.828515 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:12.830653 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:12.830659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:12.830676 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:12.830983 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:12.831326 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:12.831329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:12.831330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:12.831496 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:12.832050 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:12.832056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:12.832445 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 00:39:12.832628 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:39:12.832947 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:12.835472 4673 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34419
I20260927 00:39:12.835760 4673 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:12.836424 4673 heartbeater.cc:511] Master 127.0.98.62:34419 requested a full tablet report, sending...
I20260927 00:39:12.838047 4481 ts_manager.cc:194] Registered new tserver with Master: b4e319c9471d41a3b74acfdbde6d24e3 (127.0.98.1:35305)
I20260927 00:39:12.839212 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:12.851629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:12.968330 4678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:12.968611 4678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:12.968659 4678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:12.972877 4678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:12.972949 4678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:12.973024 4678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:12.978595 4678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:12.979717 4678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:12.981029 4678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:12.991353 4683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:12.991415 4686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:12.991326 4684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:12.991638 4678 server_base.cc:1034] running on GCE node
I20260927 00:39:12.992129 4678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:12.992844 4678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:12.994028 4678 hybrid_clock.cc:648] HybridClock initialized: now 1790469552994011 us; error 26 us; skew 500 ppm
Sep 27 00:39:12 dist-test-slave-xltq krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:12.996896 4678 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:12.997648 4678 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:12.997711 4678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:12.997905 4678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:12.999981 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "d49c12d759ad438488fbca28a8f7cad4"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "4ad531d395b49a155db935cbbe901484"
server_key_iv: "e5b51da2f7d9add6fdb3925f4dc3cc4d"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.000573 4678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d49c12d759ad438488fbca28a8f7cad4"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "4ad531d395b49a155db935cbbe901484"
server_key_iv: "e5b51da2f7d9add6fdb3925f4dc3cc4d"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.004390 4678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:39:13.007066 4692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.008364 4678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:13.008481 4678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "d49c12d759ad438488fbca28a8f7cad4"
format_stamp: "Formatted at 2026-09-27 00:39:12 on dist-test-slave-xltq"
server_key: "4ad531d395b49a155db935cbbe901484"
server_key_iv: "e5b51da2f7d9add6fdb3925f4dc3cc4d"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.008610 4678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:13.025797 4678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:13.029426 4678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:13.029628 4678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:13.030260 4678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:13.031162 4678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:13.031219 4678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.031291 4678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:13.031317 4678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.041142 4678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39863
I20260927 00:39:13.041162 4805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39863 every 8 connection(s)
I20260927 00:39:13.041846 4678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:13.048795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4678
I20260927 00:39:13.048926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:13.049286 392 external_mini_cluster.cc:1428] Setting key 60ff1bf9bf9eb03f77931fe194ba3eae
Sep 27 00:39:13 dist-test-slave-xltq krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:13.057826 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.043820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35073 (local address 127.0.98.62:34419)
0927 00:39:13.044054 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.044059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.045089 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 00:39:13.046453 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.046456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.046509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.046591 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:13.048071 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.048748 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.050346 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.050638 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.050751 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.054047 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:13.054051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:13.054053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:13.054085 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:13.055607 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.056196 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.056226 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.056228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.056310 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.056686 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.056689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.056691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.056899 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:13.056985 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.057529 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.057662 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:13.058743 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.044315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34419 (local address 127.0.98.2:35073)
0927 00:39:13.044860 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.044978 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.046223 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.046906 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.046914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.047330 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:13.047883 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.047900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.049039 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.049043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.050219 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.050232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.050827 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.050832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.051067 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.051813 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:13.051835 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:13.053792 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:13.055745 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:13.055752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:13.055772 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:13.056089 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:13.056439 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:13.056442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:13.056443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:13.056592 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:13.057000 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:13.057008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:13.057282 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.057469 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.057837 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":129,"threads_started":1}
I20260927 00:39:13.060146 4806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34419
I20260927 00:39:13.060495 4806 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:13.061048 4806 heartbeater.cc:511] Master 127.0.98.62:34419 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:13.062209 4481 ts_manager.cc:194] Registered new tserver with Master: d49c12d759ad438488fbca28a8f7cad4 (127.0.98.2:39863)
I20260927 00:39:13.062997 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:13.079773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:13.193143 4811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:13.193436 4811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:13.193482 4811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:13.197582 4811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:13.197651 4811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:13.197716 4811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:13.203223 4811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:13.204471 4811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:13.205538 4811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:13.215337 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:13.215368 4811 server_base.cc:1034] running on GCE node
W20260927 00:39:13.215279 4816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:13.215517 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:13.215987 4811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:13.216735 4811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:13.218011 4811 hybrid_clock.cc:648] HybridClock initialized: now 1790469553217991 us; error 33 us; skew 500 ppm
Sep 27 00:39:13 dist-test-slave-xltq krb5kdc[4439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:13.221319 4811 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:13.222177 4811 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:13.222254 4811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:13.222462 4811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:13.224639 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "28c590ee6b1747d3b50b845d7ea52ad7"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "b5974f581670a68d7f73b19af07b9c52"
server_key_iv: "ee48e947bbeed7f1289dff76cbdea2ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.225220 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "28c590ee6b1747d3b50b845d7ea52ad7"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "b5974f581670a68d7f73b19af07b9c52"
server_key_iv: "ee48e947bbeed7f1289dff76cbdea2ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.229270 4811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:13.231952 4825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.233381 4811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:13.233515 4811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "28c590ee6b1747d3b50b845d7ea52ad7"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "b5974f581670a68d7f73b19af07b9c52"
server_key_iv: "ee48e947bbeed7f1289dff76cbdea2ef"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.233633 4811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:13.244714 4811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:13.248510 4811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:13.248711 4811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:13.249341 4811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:13.250216 4811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:13.250265 4811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.250317 4811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:13.250346 4811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.258906 4811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33087
I20260927 00:39:13.259032 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33087 every 8 connection(s)
I20260927 00:39:13.259542 4811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:13.266512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4811
I20260927 00:39:13.266659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:13.266965 392 external_mini_cluster.cc:1428] Setting key 9fbd65723c5a8ca755599bb0da51b678
Sep 27 00:39:13 dist-test-slave-xltq krb5kdc[4439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:13.274891 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.261478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47561 (local address 127.0.98.62:34419)
0927 00:39:13.261658 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.261662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.262490 (+ 828us) server_negotiation.cc:407] Connection header received
0927 00:39:13.263848 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.263850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.263899 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.263977 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:13.265485 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.266028 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.267762 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.268167 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.268302 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.271079 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:13.271082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:13.271084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:13.271113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:13.272536 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.273041 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.273044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.273045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.273125 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.273570 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.273574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.273576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.273890 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:13.273990 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.274569 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.274685 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:39:13.275713 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.261868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34419 (local address 127.0.98.3:47561)
0927 00:39:13.262327 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.262369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.263589 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.264256 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.264265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.264697 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:13.265303 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.265321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.266213 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.266217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.267584 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.267595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.268354 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.268358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.268590 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.269207 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:13.269227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:13.270865 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:13.272641 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:13.272646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:13.272661 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:13.272943 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:13.273240 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:13.273244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:13.273246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:13.273447 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:13.274019 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:13.274025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:13.274301 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.274486 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.274770 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260927 00:39:13.276960 4939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34419
I20260927 00:39:13.277205 4939 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:13.277680 4939 heartbeater.cc:511] Master 127.0.98.62:34419 requested a full tablet report, sending...
I20260927 00:39:13.278678 4481 ts_manager.cc:194] Registered new tserver with Master: 28c590ee6b1747d3b50b845d7ea52ad7 (127.0.98.3:33087)
I20260927 00:39:13.279240 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47561
I20260927 00:39:13.281646 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:39:13.291455 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.283586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.0.98.62:34419)
0927 00:39:13.283775 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.283779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.283889 (+ 110us) server_negotiation.cc:407] Connection header received
0927 00:39:13.284027 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.284031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.284076 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.284142 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:13.284942 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.285468 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.286295 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.286518 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.286616 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.288035 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:13.288038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:13.288040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:13.288068 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:13.289603 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.290097 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.290101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.290103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.290186 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:13.290445 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:13.290448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:13.290450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:13.290689 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:13.290807 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.291157 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.291289 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 00:39:13.294767 4481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:13.298072 4481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:13.313000 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.307833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.0.98.3:33087)
0927 00:39:13.308559 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.308567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.308582 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:13.308636 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.308640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.308814 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.308953 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:13.309304 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310055 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.311519 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.312182 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.312309 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.312462 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.312706 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.312818 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":108,"threads_started":1}
I20260927 00:39:13.312997 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.307696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33087 (local address 127.0.0.1:42340)
0927 00:39:13.308072 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.308128 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.308302 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.309091 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.309093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.309106 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:13.309187 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.309193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310175 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.310178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.311399 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.311409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.312316 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.312320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.312490 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.312536 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.312658 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.312799 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":21,"spinlock_wait_cycles":10240,"thread_start_us":81,"threads_started":1}
I20260927 00:39:13.313820 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.307004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.98.2:39863)
0927 00:39:13.307727 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.307740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.308242 (+ 502us) server_negotiation.cc:407] Connection header received
0927 00:39:13.308457 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.308462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.308633 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.308751 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:13.309070 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310287 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.311964 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.312785 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.312894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.313036 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.313577 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.313677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":113,"threads_started":1}
I20260927 00:39:13.313838 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.307146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35305 (local address 127.0.0.1:55430)
0927 00:39:13.308073 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.308129 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.308302 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.309214 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.309217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.309231 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:13.309310 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.309316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310558 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.310561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.311819 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.311826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.313104 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.313108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.313540 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.313565 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.313634 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.313705 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":136,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20260927 00:39:13.314312 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.307318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.0.98.1:35305)
0927 00:39:13.307924 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.307930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.308249 (+ 319us) server_negotiation.cc:407] Connection header received
0927 00:39:13.308927 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.308932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.309085 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.309232 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:13.309381 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310433 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.311947 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.312720 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.312869 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.313035 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.314071 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.314171 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":80,"threads_started":1}
I20260927 00:39:13.314577 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.306841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39863 (local address 127.0.0.1:43810)
0927 00:39:13.308116 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.308130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.308302 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.308772 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.308778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.308798 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:13.308962 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.308970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.310438 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.310441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.311814 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.311826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.313099 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.313304 (+ 205us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.313416 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.313442 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.314379 (+ 937us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.314454 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":112,"thread_start_us":173,"threads_started":1}
I20260927 00:39:13.316524 4873 tablet_service.cc:1467] Processing CreateTablet for tablet 4029c47653f241b2ae1b964076daaf4e (DEFAULT_TABLE table=test-table [id=962b2f8f88874fe79668ceb501116731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:13.316915 4607 tablet_service.cc:1467] Processing CreateTablet for tablet 4029c47653f241b2ae1b964076daaf4e (DEFAULT_TABLE table=test-table [id=962b2f8f88874fe79668ceb501116731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:13.317279 4740 tablet_service.cc:1467] Processing CreateTablet for tablet 4029c47653f241b2ae1b964076daaf4e (DEFAULT_TABLE table=test-table [id=962b2f8f88874fe79668ceb501116731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:13.317709 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4029c47653f241b2ae1b964076daaf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:13.317901 4607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4029c47653f241b2ae1b964076daaf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:13.318217 4740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4029c47653f241b2ae1b964076daaf4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:13.324193 4956 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: Bootstrap starting.
I20260927 00:39:13.325598 4957 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Bootstrap starting.
I20260927 00:39:13.326505 4956 tablet_bootstrap.cc:654] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:13.327481 4957 tablet_bootstrap.cc:654] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:13.327567 4956 log.cc:824] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:13.328356 4958 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Bootstrap starting.
I20260927 00:39:13.328672 4957 log.cc:824] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:13.329761 4956 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: No bootstrap required, opened a new log
I20260927 00:39:13.329967 4956 ts_tablet_manager.cc:1397] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:13.330410 4958 tablet_bootstrap.cc:654] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:13.330827 4957 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: No bootstrap required, opened a new log
I20260927 00:39:13.331035 4957 ts_tablet_manager.cc:1397] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:13.331422 4958 log.cc:824] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:13.333132 4958 tablet_bootstrap.cc:492] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: No bootstrap required, opened a new log
I20260927 00:39:13.333304 4958 ts_tablet_manager.cc:1397] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:39:13.333932 4956 raft_consensus.cc:348] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.334221 4956 raft_consensus.cc:374] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:13.334313 4956 raft_consensus.cc:729] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e319c9471d41a3b74acfdbde6d24e3, State: Initialized, Role: FOLLOWER
I20260927 00:39:13.334862 4956 consensus_queue.cc:260] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.335259 4957 raft_consensus.cc:348] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.335580 4957 raft_consensus.cc:374] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:13.335662 4957 raft_consensus.cc:729] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49c12d759ad438488fbca28a8f7cad4, State: Initialized, Role: FOLLOWER
I20260927 00:39:13.335915 4956 ts_tablet_manager.cc:1428] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:39:13.336172 4957 consensus_queue.cc:260] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.336334 4673 heartbeater.cc:503] Master 127.0.98.62:34419 was elected leader, sending a full tablet report...
I20260927 00:39:13.336959 4806 heartbeater.cc:503] Master 127.0.98.62:34419 was elected leader, sending a full tablet report...
I20260927 00:39:13.336779 4958 raft_consensus.cc:348] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.337047 4958 raft_consensus.cc:374] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:13.337091 4958 raft_consensus.cc:729] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c590ee6b1747d3b50b845d7ea52ad7, State: Initialized, Role: FOLLOWER
I20260927 00:39:13.337212 4957 ts_tablet_manager.cc:1428] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:39:13.337541 4958 consensus_queue.cc:260] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.338613 4958 ts_tablet_manager.cc:1428] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:39:13.338891 4962 raft_consensus.cc:482] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:13.339015 4962 raft_consensus.cc:504] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.339802 4939 heartbeater.cc:503] Master 127.0.98.62:34419 was elected leader, sending a full tablet report...
I20260927 00:39:13.341521 4962 leader_election.cc:288] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49c12d759ad438488fbca28a8f7cad4 (127.0.98.2:39863), 28c590ee6b1747d3b50b845d7ea52ad7 (127.0.98.3:33087)
I20260927 00:39:13.344833 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.340917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59331 (local address 127.0.98.2:39863)
0927 00:39:13.341050 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.341054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.341072 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:13.341115 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.341118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.341164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.341234 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:13.341607 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.342286 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.343582 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.344188 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.344307 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.344372 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.344655 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.344730 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:39:13.344848 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.340464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39863 (local address 127.0.98.1:59331)
0927 00:39:13.340739 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.340754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.340850 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.341350 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.341353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.341371 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:13.341490 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.341497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.342374 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.342377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.343442 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.343453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.344325 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.344329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.344481 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.344509 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.344623 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.344698 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":104,"threads_started":1}
I20260927 00:39:13.345381 4760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4029c47653f241b2ae1b964076daaf4e" candidate_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49c12d759ad438488fbca28a8f7cad4" is_pre_election: true
I20260927 00:39:13.345480 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.341255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33087 (local address 127.0.98.1:41771)
0927 00:39:13.341809 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.341822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.341921 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.342217 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.342220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.342235 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:13.342332 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.342338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.343167 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.343170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.344373 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:13.344383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.345146 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.345149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:13.345244 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.345262 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.345320 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.345368 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":87,"threads_started":1}
I20260927 00:39:13.345537 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.341565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41771 (local address 127.0.98.3:33087)
0927 00:39:13.341659 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.341663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.341904 (+ 241us) server_negotiation.cc:407] Connection header received
0927 00:39:13.342026 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.342029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.342072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.342130 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:13.342489 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.343052 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.344463 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.344942 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.345038 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.345090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.345355 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.345429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":27}
I20260927 00:39:13.345713 4760 raft_consensus.cc:2454] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e319c9471d41a3b74acfdbde6d24e3 in term 0.
I20260927 00:39:13.346161 4563 leader_election.cc:302] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e319c9471d41a3b74acfdbde6d24e3, d49c12d759ad438488fbca28a8f7cad4; no voters:
I20260927 00:39:13.346390 4893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4029c47653f241b2ae1b964076daaf4e" candidate_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" is_pre_election: true
I20260927 00:39:13.346472 4962 raft_consensus.cc:2781] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:13.346565 4962 raft_consensus.cc:482] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:13.346614 4962 raft_consensus.cc:3037] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:13.346650 4893 raft_consensus.cc:2454] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e319c9471d41a3b74acfdbde6d24e3 in term 0.
I20260927 00:39:13.347733 4962 raft_consensus.cc:504] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.348060 4962 leader_election.cc:288] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [CANDIDATE]: Term 1 election: Requested vote from peers d49c12d759ad438488fbca28a8f7cad4 (127.0.98.2:39863), 28c590ee6b1747d3b50b845d7ea52ad7 (127.0.98.3:33087)
I20260927 00:39:13.348395 4760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4029c47653f241b2ae1b964076daaf4e" candidate_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49c12d759ad438488fbca28a8f7cad4"
I20260927 00:39:13.348456 4893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4029c47653f241b2ae1b964076daaf4e" candidate_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c590ee6b1747d3b50b845d7ea52ad7"
I20260927 00:39:13.348510 4760 raft_consensus.cc:3037] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:13.348591 4893 raft_consensus.cc:3037] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:13.349568 4893 raft_consensus.cc:2454] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e319c9471d41a3b74acfdbde6d24e3 in term 1.
I20260927 00:39:13.349576 4760 raft_consensus.cc:2454] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e319c9471d41a3b74acfdbde6d24e3 in term 1.
I20260927 00:39:13.349831 4563 leader_election.cc:302] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e319c9471d41a3b74acfdbde6d24e3, d49c12d759ad438488fbca28a8f7cad4; no voters:
I20260927 00:39:13.350008 4962 raft_consensus.cc:2781] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:13.350108 4962 raft_consensus.cc:686] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [term 1 LEADER]: Becoming Leader. State: Replica: b4e319c9471d41a3b74acfdbde6d24e3, State: Running, Role: LEADER
I20260927 00:39:13.350409 4962 consensus_queue.cc:237] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } }
I20260927 00:39:13.353648 4483 catalog_manager.cc:5495] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4e319c9471d41a3b74acfdbde6d24e3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e319c9471d41a3b74acfdbde6d24e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35305 } health_report { overall_health: HEALTHY } } }
I20260927 00:39:13.376308 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.372780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.98.1:35305)
0927 00:39:13.372893 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.372896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.372909 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:13.373061 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.373064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.373121 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.373202 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:13.373452 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.374018 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.374871 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:13.375122 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:13.375202 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:13.375464 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:13.375886 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:13.376029 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.376094 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.376144 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:39:13.382251 4893 raft_consensus.cc:1261] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 1 FOLLOWER]: Refusing update from remote peer b4e319c9471d41a3b74acfdbde6d24e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:13.382251 4760 raft_consensus.cc:1261] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 1 FOLLOWER]: Refusing update from remote peer b4e319c9471d41a3b74acfdbde6d24e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:13.382958 4965 consensus_queue.cc:1035] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49c12d759ad438488fbca28a8f7cad4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:13.383162 4962 consensus_queue.cc:1035] T 4029c47653f241b2ae1b964076daaf4e P b4e319c9471d41a3b74acfdbde6d24e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c590ee6b1747d3b50b845d7ea52ad7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:13.391949 4971 mvcc.cc:205] Tried to move back new op lower bound from 7333763290646761472 to 7333763290525949952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763290646761472}]
I20260927 00:39:13.392856 4972 mvcc.cc:205] Tried to move back new op lower bound from 7333763290646761472 to 7333763290525949952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763290646761472}]
I20260927 00:39:13.399912 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:13.400128 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:13.402861 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 859b63d2d6cb41119979d5084e826861: Sending DeleteTablet for 3 replicas of tablet 4029c47653f241b2ae1b964076daaf4e
I20260927 00:39:13.403790 4607 tablet_service.cc:1514] Processing DeleteTablet for tablet 4029c47653f241b2ae1b964076daaf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55430
I20260927 00:39:13.403849 4740 tablet_service.cc:1514] Processing DeleteTablet for tablet 4029c47653f241b2ae1b964076daaf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43810
I20260927 00:39:13.404450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4545
I20260927 00:39:13.404597 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet 4029c47653f241b2ae1b964076daaf4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42340
I20260927 00:39:13.405444 4981 tablet_replica.cc:330] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: stopping tablet replica
I20260927 00:39:13.405879 4981 raft_consensus.cc:2229] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:13.406317 4981 raft_consensus.cc:2258] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:13.406631 4982 tablet_replica.cc:330] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: stopping tablet replica
I20260927 00:39:13.406952 4982 raft_consensus.cc:2229] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:13.407234 4982 raft_consensus.cc:2258] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:13.408058 4981 ts_tablet_manager.cc:1905] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:13.408483 4982 ts_tablet_manager.cc:1905] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:13.410701 4982 ts_tablet_manager.cc:1918] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:13.410794 4982 log.cc:1196] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/4029c47653f241b2ae1b964076daaf4e
I20260927 00:39:13.410882 4981 ts_tablet_manager.cc:1918] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:13.411016 4981 log.cc:1196] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/4029c47653f241b2ae1b964076daaf4e
I20260927 00:39:13.411137 4982 ts_tablet_manager.cc:1939] T 4029c47653f241b2ae1b964076daaf4e P 28c590ee6b1747d3b50b845d7ea52ad7: Deleting consensus metadata
I20260927 00:39:13.411331 4981 ts_tablet_manager.cc:1939] T 4029c47653f241b2ae1b964076daaf4e P d49c12d759ad438488fbca28a8f7cad4: Deleting consensus metadata
W20260927 00:39:13.411677 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35305 (error 108)
I20260927 00:39:13.412643 4470 catalog_manager.cc:4841] TS 28c590ee6b1747d3b50b845d7ea52ad7 (127.0.98.3:33087): tablet 4029c47653f241b2ae1b964076daaf4e (table test-table [id=962b2f8f88874fe79668ceb501116731]) successfully deleted
I20260927 00:39:13.412791 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.412505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35305 (local address 127.0.0.1:55452)
0927 00:39:13.412636 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.412702 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:39:13.412818 4470 catalog_manager.cc:4841] TS d49c12d759ad438488fbca28a8f7cad4 (127.0.98.2:39863): tablet 4029c47653f241b2ae1b964076daaf4e (table test-table [id=962b2f8f88874fe79668ceb501116731]) successfully deleted
I20260927 00:39:13.412861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4678
W20260927 00:39:13.412993 4469 catalog_manager.cc:4584] TS b4e319c9471d41a3b74acfdbde6d24e3 (127.0.98.1:35305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4029c47653f241b2ae1b964076daaf4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35305: connect: Connection refused (error 111)
I20260927 00:39:13.417665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4811
I20260927 00:39:13.422700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4452
2026-09-27T00:39:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2234 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:13Z Disabled control of system clock
I20260927 00:39:13.451035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:39:13.562599 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:13.562907 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:13.562960 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:13.567260 4987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:13.567343 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:13.567364 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:13.567385 4987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:13.567405 4987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:13.572499 4987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:13.573639 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:13.575086 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:13.583911 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:13.584141 4987 server_base.cc:1034] running on GCE node
W20260927 00:39:13.583933 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:13.583953 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:13.584864 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:13.586093 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:13.587278 4987 hybrid_clock.cc:648] HybridClock initialized: now 1790469553587240 us; error 57 us; skew 500 ppm
I20260927 00:39:13.589331 4987 webserver.cc:460] Webserver started at http://127.0.98.62:34443/ using document root <none> and password file <none>
I20260927 00:39:13.589947 4987 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:13.590024 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:13.590227 4987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:13.592408 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "b84488bf1a7f41b7ae653d4a8567d9c3"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "2a6796b6d045e08291c53996191b220d"
server_key_iv: "f0535c3de68325c30235bf6a22c958c9"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.592980 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "b84488bf1a7f41b7ae653d4a8567d9c3"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "2a6796b6d045e08291c53996191b220d"
server_key_iv: "f0535c3de68325c30235bf6a22c958c9"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.597038 4987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:39:13.599745 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.601223 4987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:39:13.601366 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "b84488bf1a7f41b7ae653d4a8567d9c3"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "2a6796b6d045e08291c53996191b220d"
server_key_iv: "f0535c3de68325c30235bf6a22c958c9"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.601492 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:13.606384 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:13.607074 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:13.607260 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:13.614230 4987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37747
I20260927 00:39:13.614248 5053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37747 every 8 connection(s)
I20260927 00:39:13.615223 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:13.616762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 4987
I20260927 00:39:13.616870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:13.617138 392 external_mini_cluster.cc:1428] Setting key 004dbc9cfa6fcaa8bbef13bc33310827
I20260927 00:39:13.618943 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:13.622670 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.618434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.0.98.62:37747)
0927 00:39:13.619078 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.619094 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.619131 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:39:13.619814 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.619832 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.620231 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.620559 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:13.621375 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:13.621379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:13.621391 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:13.621399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:13.621451 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:13.621498 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:13.621610 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.621699 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.621931 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260927 00:39:13.625185 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: Bootstrap starting.
I20260927 00:39:13.627430 5054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:13.628244 5054 log.cc:824] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:13.630386 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: No bootstrap required, opened a new log
I20260927 00:39:13.634263 5054 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 00:39:13.634522 5054 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:13.634574 5054 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84488bf1a7f41b7ae653d4a8567d9c3, State: Initialized, Role: FOLLOWER
I20260927 00:39:13.635052 5054 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 00:39:13.635193 5054 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:13.635267 5054 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:13.635370 5054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:13.636430 5054 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 00:39:13.636824 5054 leader_election.cc:302] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84488bf1a7f41b7ae653d4a8567d9c3; no voters:
I20260927 00:39:13.637185 5054 leader_election.cc:288] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:13.637316 5059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:13.637545 5059 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [term 1 LEADER]: Becoming Leader. State: Replica: b84488bf1a7f41b7ae653d4a8567d9c3, State: Running, Role: LEADER
I20260927 00:39:13.637897 5059 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } }
I20260927 00:39:13.638458 5054 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:13.639569 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84488bf1a7f41b7ae653d4a8567d9c3. Latest consensus state: current_term: 1 leader_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260927 00:39:13.639715 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:13.640027 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84488bf1a7f41b7ae653d4a8567d9c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37747 } } }
I20260927 00:39:13.640138 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:13.640175 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:13.643213 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:13.648784 5068 catalog_manager.cc:1347] Generated new cluster ID: 9b292b048b184e28921ff6b8b78e9102
I20260927 00:39:13.648874 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:13.661497 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:13.661927 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:13.672319 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: Generated new TSK 0
I20260927 00:39:13.673206 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:13.689005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--builtin_ntp_servers=127.0.98.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:39:13.805960 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:13.806253 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:13.806294 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:13.810611 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:13.810688 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:13.810756 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:13.816087 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:13.817301 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:13.818760 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:13.828795 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:13.828806 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:13.828799 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:13.829178 5078 server_base.cc:1034] running on GCE node
I20260927 00:39:13.829775 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:13.830535 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:13.831727 5078 hybrid_clock.cc:648] HybridClock initialized: now 1790469553831704 us; error 42 us; skew 500 ppm
I20260927 00:39:13.839542 5078 webserver.cc:460] Webserver started at http://127.0.98.1:35865/ using document root <none> and password file <none>
I20260927 00:39:13.840291 5078 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:13.840382 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:13.840606 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:13.842841 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ba388cc91ba4d5eb8d8a370e34d5217"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "0021bcebecb6a5a2beb943cc9377806f"
server_key_iv: "0d1039544b1f2b44e8443a336ba11c31"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.843457 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba388cc91ba4d5eb8d8a370e34d5217"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "0021bcebecb6a5a2beb943cc9377806f"
server_key_iv: "0d1039544b1f2b44e8443a336ba11c31"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.847649 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 00:39:13.850497 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.851913 5078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:13.852110 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "0ba388cc91ba4d5eb8d8a370e34d5217"
format_stamp: "Formatted at 2026-09-27 00:39:13 on dist-test-slave-xltq"
server_key: "0021bcebecb6a5a2beb943cc9377806f"
server_key_iv: "0d1039544b1f2b44e8443a336ba11c31"
server_key_version: "encryptionkey@0"
I20260927 00:39:13.852269 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:13.857081 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:13.857733 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:13.857901 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:13.858619 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:13.859614 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:13.859666 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.859720 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:13.859748 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:13.868548 5078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39315
I20260927 00:39:13.868607 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39315 every 8 connection(s)
I20260927 00:39:13.869550 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:13.876050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5078
I20260927 00:39:13.876187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:13.876526 392 external_mini_cluster.cc:1428] Setting key 2a0b96c1c69c8f88949369e6b95daa45
I20260927 00:39:13.876521 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.871621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57753 (local address 127.0.98.62:37747)
0927 00:39:13.871756 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:13.871760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:13.872951 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 00:39:13.874436 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:13.874440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:13.874484 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:13.874580 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:13.875859 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:13.875863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:13.875865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:13.875883 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:13.875906 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:13.875915 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:13.876004 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:13.876349 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:39:13.876404 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:39:13.877643 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:13.871975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.1:57753)
0927 00:39:13.872759 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:13.872809 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:13.874154 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:13.874763 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:13.874771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:13.874876 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:13.875553 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:13.875554 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:13.875603 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:13.875682 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:13.876074 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:13.876087 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:13.876093 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:13.876341 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 00:39:13.876653 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":144,"threads_started":1}
I20260927 00:39:13.878793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--builtin_ntp_servers=127.0.98.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:39:13.878957 5206 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 00:39:13.879251 5206 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:13.879828 5206 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 00:39:13.881703 5017 ts_manager.cc:194] Registered new tserver with Master: 0ba388cc91ba4d5eb8d8a370e34d5217 (127.0.98.1:39315)
W20260927 00:39:13.994933 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:13.995253 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:13.995328 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:13.999857 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:13.999964 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:14.000056 5209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:14.005244 5209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:14.006407 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:14.007908 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:14.017972 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.018083 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.017972 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.018276 5209 server_base.cc:1034] running on GCE node
I20260927 00:39:14.018715 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:14.019440 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:14.020589 5209 hybrid_clock.cc:648] HybridClock initialized: now 1790469554020566 us; error 34 us; skew 500 ppm
I20260927 00:39:14.022704 5209 webserver.cc:460] Webserver started at http://127.0.98.2:44563/ using document root <none> and password file <none>
I20260927 00:39:14.023300 5209 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:14.023375 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:14.023590 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:14.025833 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "b782eae4403940718c25c07c389fb625"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "7efd262fdc75eeef5df8b7bb9e6b8ffa"
server_key_iv: "6f8b815bb2daf71e760a4670178110bf"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.026417 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b782eae4403940718c25c07c389fb625"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "7efd262fdc75eeef5df8b7bb9e6b8ffa"
server_key_iv: "6f8b815bb2daf71e760a4670178110bf"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.030565 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:14.033345 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.034850 5209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:14.034979 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "b782eae4403940718c25c07c389fb625"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "7efd262fdc75eeef5df8b7bb9e6b8ffa"
server_key_iv: "6f8b815bb2daf71e760a4670178110bf"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.035118 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:14.040273 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:14.041199 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:14.041371 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:14.041925 5209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:14.042939 5209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:14.042989 5209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.043053 5209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:14.043078 5209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.052121 5209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33161
I20260927 00:39:14.052157 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33161 every 8 connection(s)
I20260927 00:39:14.053138 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:14.055763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5209
I20260927 00:39:14.055896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:14.056195 392 external_mini_cluster.cc:1428] Setting key 54d70c05f65fc4c577d29d91b441a5d0
I20260927 00:39:14.058200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--builtin_ntp_servers=127.0.98.20:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 00:39:14.059947 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.055020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51513 (local address 127.0.98.62:37747)
0927 00:39:14.055196 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.055199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.056736 (+ 1537us) server_negotiation.cc:407] Connection header received
0927 00:39:14.058161 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.058165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.058207 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.058283 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.059461 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.059465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.059468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.059484 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.059510 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.059520 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.059605 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.059796 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.059840 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 00:39:14.061148 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.055556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.2:51513)
0927 00:39:14.056540 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.056609 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.057949 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.058468 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.058476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.058586 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.059138 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.059140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.059183 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.059259 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.059603 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.059611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.059616 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.059809 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.060089 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":143,"threads_started":1}
I20260927 00:39:14.062347 5337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 00:39:14.062606 5337 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:14.063025 5337 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 00:39:14.064105 5017 ts_manager.cc:194] Registered new tserver with Master: b782eae4403940718c25c07c389fb625 (127.0.98.2:33161)
W20260927 00:39:14.175066 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:14.175357 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:14.175398 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:14.179615 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:14.179697 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:14.179759 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:14.184768 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:14.185873 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:14.187265 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:14.197513 5340 server_base.cc:1034] running on GCE node
W20260927 00:39:14.197461 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.197453 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.197450 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.198158 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:14.198791 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:14.199976 5340 hybrid_clock.cc:648] HybridClock initialized: now 1790469554199957 us; error 31 us; skew 500 ppm
I20260927 00:39:14.202102 5340 webserver.cc:460] Webserver started at http://127.0.98.3:41277/ using document root <none> and password file <none>
I20260927 00:39:14.202793 5340 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:14.202869 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:14.203084 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:14.205348 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "e0048e4c149340800ae289bcaf0bde50"
server_key_iv: "9d40d62a0cea889eb1c83dfcfe62013a"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.205945 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "e0048e4c149340800ae289bcaf0bde50"
server_key_iv: "9d40d62a0cea889eb1c83dfcfe62013a"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.209878 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:14.212469 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.213799 5340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:39:14.213923 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "e0048e4c149340800ae289bcaf0bde50"
server_key_iv: "9d40d62a0cea889eb1c83dfcfe62013a"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.214061 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:14.218899 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:14.219539 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:14.219705 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:14.220294 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:14.221249 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:14.221307 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.221364 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:14.221401 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.230011 5340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43119
I20260927 00:39:14.230036 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43119 every 8 connection(s)
I20260927 00:39:14.230938 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:14.235023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5340
I20260927 00:39:14.235141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:14.235417 392 external_mini_cluster.cc:1428] Setting key ca2ea4663eb96aaa20c8a3968521f47a
I20260927 00:39:14.237382 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.232762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52569 (local address 127.0.98.62:37747)
0927 00:39:14.232897 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.232901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.234060 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 00:39:14.235470 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.235473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.235520 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.235606 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.236834 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.236838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.236840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.236852 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.236874 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.236884 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.237037 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.237196 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.237252 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:39:14.238301 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.233192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37747 (local address 127.0.98.3:52569)
0927 00:39:14.233829 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.233869 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.235210 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.235831 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.235840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.235961 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.236531 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.236533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.236586 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.236683 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.237021 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.237027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.237031 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.237220 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.237465 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260927 00:39:14.239563 5468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37747
I20260927 00:39:14.239774 5468 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:14.240286 5468 heartbeater.cc:511] Master 127.0.98.62:37747 requested a full tablet report, sending...
I20260927 00:39:14.241343 5017 ts_manager.cc:194] Registered new tserver with Master: 719b00ea5a854c5e8ecb3ef6b40d3c86 (127.0.98.3:43119)
I20260927 00:39:14.249902 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:39:14.253840 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.252317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.62:37747)
0927 00:39:14.252448 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.252452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.252509 (+ 57us) server_negotiation.cc:407] Connection header received
0927 00:39:14.252655 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.252658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.252696 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.252765 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.253454 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.253458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.253461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.253473 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.253494 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.253502 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.253591 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.253660 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.253704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:39:14.257650 5017 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:14.260805 5017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:14.273557 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.270249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33161 (local address 127.0.0.1:40416)
0927 00:39:14.271117 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.271173 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.271307 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.272029 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.272036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.272118 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.272728 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.272730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.272751 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.272819 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.273198 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.273207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.273213 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.273323 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.273398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"spinlock_wait_cycles":4736,"thread_start_us":98,"threads_started":1}
I20260927 00:39:14.273591 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.270512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.2:33161)
0927 00:39:14.271557 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.271569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.271593 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:39:14.271648 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.271654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.271867 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.272008 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.272963 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.272970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.272983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.273017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.273064 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.273104 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.273225 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.273348 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.273437 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":123,"threads_started":1}
I20260927 00:39:14.273614 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.270552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39315 (local address 127.0.0.1:41520)
0927 00:39:14.271525 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.271537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.271605 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.272428 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.272432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.272446 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.272858 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.272860 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.272872 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.272900 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.273350 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.273353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.273356 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.273438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.273476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":97,"threads_started":1}
I20260927 00:39:14.273696 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.271104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.0.98.1:39315)
0927 00:39:14.271560 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.271572 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.271595 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:39:14.271717 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.271722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.271867 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.272008 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.273180 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.273186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.273198 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.273208 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.273249 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.273274 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.273380 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.273454 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.273534 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":103,"threads_started":1}
I20260927 00:39:14.274919 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.270319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43119 (local address 127.0.0.1:42848)
0927 00:39:14.271169 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.271186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.271307 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.272658 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.272661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.272936 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.274265 (+ 1329us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.274267 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.274278 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.274304 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.274694 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.274697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.274700 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.274768 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.274806 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"spinlock_wait_cycles":8832,"thread_start_us":119,"threads_started":1}
I20260927 00:39:14.275453 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.270887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.0.98.3:43119)
0927 00:39:14.271791 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.271802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.271824 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:39:14.271874 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.271880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.272069 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.272196 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.274440 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.274446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.274459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.274473 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.274517 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.274555 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.274653 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.275110 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.275192 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":167,"threads_started":1}
I20260927 00:39:14.277225 5271 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 (DEFAULT_TABLE table=test-table [id=ae9ed1f8899b4836854aa419efcbac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:14.277225 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 (DEFAULT_TABLE table=test-table [id=ae9ed1f8899b4836854aa419efcbac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:14.278424 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:14.278424 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:14.278383 5402 tablet_service.cc:1467] Processing CreateTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 (DEFAULT_TABLE table=test-table [id=ae9ed1f8899b4836854aa419efcbac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:14.279285 5402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:14.285518 5485 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Bootstrap starting.
I20260927 00:39:14.286955 5487 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: Bootstrap starting.
I20260927 00:39:14.288538 5485 tablet_bootstrap.cc:654] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:14.288604 5486 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Bootstrap starting.
I20260927 00:39:14.289422 5487 tablet_bootstrap.cc:654] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:14.289492 5485 log.cc:824] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:14.290251 5487 log.cc:824] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:14.291170 5486 tablet_bootstrap.cc:654] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:14.291934 5485 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: No bootstrap required, opened a new log
I20260927 00:39:14.291941 5486 log.cc:824] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:14.292165 5485 ts_tablet_manager.cc:1397] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:39:14.292337 5487 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: No bootstrap required, opened a new log
I20260927 00:39:14.292519 5487 ts_tablet_manager.cc:1397] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:39:14.294180 5486 tablet_bootstrap.cc:492] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: No bootstrap required, opened a new log
I20260927 00:39:14.294375 5486 ts_tablet_manager.cc:1397] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:14.297001 5487 raft_consensus.cc:348] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.297001 5485 raft_consensus.cc:348] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.297336 5487 raft_consensus.cc:374] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:14.297335 5485 raft_consensus.cc:374] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:14.297398 5485 raft_consensus.cc:729] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719b00ea5a854c5e8ecb3ef6b40d3c86, State: Initialized, Role: FOLLOWER
I20260927 00:39:14.297398 5487 raft_consensus.cc:729] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba388cc91ba4d5eb8d8a370e34d5217, State: Initialized, Role: FOLLOWER
I20260927 00:39:14.297917 5485 consensus_queue.cc:260] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.297917 5487 consensus_queue.cc:260] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.297912 5486 raft_consensus.cc:348] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.298159 5486 raft_consensus.cc:374] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:14.298211 5486 raft_consensus.cc:729] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b782eae4403940718c25c07c389fb625, State: Initialized, Role: FOLLOWER
I20260927 00:39:14.298681 5486 consensus_queue.cc:260] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.298784 5206 heartbeater.cc:503] Master 127.0.98.62:37747 was elected leader, sending a full tablet report...
I20260927 00:39:14.299050 5487 ts_tablet_manager.cc:1428] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:39:14.299050 5485 ts_tablet_manager.cc:1428] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:39:14.299407 5468 heartbeater.cc:503] Master 127.0.98.62:37747 was elected leader, sending a full tablet report...
I20260927 00:39:14.299857 5337 heartbeater.cc:503] Master 127.0.98.62:37747 was elected leader, sending a full tablet report...
I20260927 00:39:14.300869 5486 ts_tablet_manager.cc:1428] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260927 00:39:14.304219 5338 tablet.cc:2369] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:39:14.333597 5491 raft_consensus.cc:482] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:14.333810 5491 raft_consensus.cc:504] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.334937 5491 leader_election.cc:288] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b782eae4403940718c25c07c389fb625 (127.0.98.2:33161), 719b00ea5a854c5e8ecb3ef6b40d3c86 (127.0.98.3:43119)
I20260927 00:39:14.336653 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.335152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33161 (local address 127.0.98.1:43359)
0927 00:39:14.335292 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.335308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.335392 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.335640 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.335643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.335657 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.336143 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.336144 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.336154 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.336181 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.336452 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.336455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.336459 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.336527 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.336557 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:39:14.336660 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.335266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43359 (local address 127.0.98.2:33161)
0927 00:39:14.335388 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.335392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.335408 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:39:14.335481 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.335484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.335523 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.335594 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.336328 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.336331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.336333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.336344 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.336363 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.336371 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.336445 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.336526 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.336561 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:39:14.337301 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.335152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43119 (local address 127.0.98.1:45397)
0927 00:39:14.335622 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.335634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.335712 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.335951 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.335955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.335971 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:14.336800 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:14.336802 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:14.336811 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:14.336837 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:14.337088 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:14.337091 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:14.337094 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:39:14.337164 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:39:14.337192 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":124,"thread_start_us":127,"threads_started":1}
I20260927 00:39:14.337327 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.335273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45397 (local address 127.0.98.3:43119)
0927 00:39:14.335408 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.335412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.335735 (+ 323us) server_negotiation.cc:407] Connection header received
0927 00:39:14.335782 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.335785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.335820 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.335901 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.336940 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.336944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.336946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.336960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.336980 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.336990 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.337071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.337175 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.337213 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:39:14.337271 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbf18fb320b4bc59b0cf3adec11a4a7" candidate_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b782eae4403940718c25c07c389fb625" is_pre_election: true
I20260927 00:39:14.337597 5291 raft_consensus.cc:2454] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba388cc91ba4d5eb8d8a370e34d5217 in term 0.
I20260927 00:39:14.338063 5095 leader_election.cc:302] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba388cc91ba4d5eb8d8a370e34d5217, b782eae4403940718c25c07c389fb625; no voters:
I20260927 00:39:14.338061 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbf18fb320b4bc59b0cf3adec11a4a7" candidate_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" is_pre_election: true
I20260927 00:39:14.338318 5422 raft_consensus.cc:2454] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba388cc91ba4d5eb8d8a370e34d5217 in term 0.
I20260927 00:39:14.338330 5491 raft_consensus.cc:2781] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:14.338408 5491 raft_consensus.cc:482] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:14.338466 5491 raft_consensus.cc:3037] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:14.339609 5491 raft_consensus.cc:504] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.339948 5491 leader_election.cc:288] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [CANDIDATE]: Term 1 election: Requested vote from peers b782eae4403940718c25c07c389fb625 (127.0.98.2:33161), 719b00ea5a854c5e8ecb3ef6b40d3c86 (127.0.98.3:43119)
I20260927 00:39:14.340366 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbf18fb320b4bc59b0cf3adec11a4a7" candidate_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86"
I20260927 00:39:14.340426 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dbf18fb320b4bc59b0cf3adec11a4a7" candidate_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b782eae4403940718c25c07c389fb625"
I20260927 00:39:14.340512 5422 raft_consensus.cc:3037] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:14.340554 5291 raft_consensus.cc:3037] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:14.341548 5291 raft_consensus.cc:2454] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba388cc91ba4d5eb8d8a370e34d5217 in term 1.
I20260927 00:39:14.341624 5422 raft_consensus.cc:2454] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba388cc91ba4d5eb8d8a370e34d5217 in term 1.
I20260927 00:39:14.341813 5095 leader_election.cc:302] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba388cc91ba4d5eb8d8a370e34d5217, b782eae4403940718c25c07c389fb625; no voters:
I20260927 00:39:14.342015 5491 raft_consensus.cc:2781] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:14.342211 5491 raft_consensus.cc:686] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba388cc91ba4d5eb8d8a370e34d5217, State: Running, Role: LEADER
I20260927 00:39:14.342562 5491 consensus_queue.cc:237] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } }
I20260927 00:39:14.345717 5015 catalog_manager.cc:5495] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ba388cc91ba4d5eb8d8a370e34d5217 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba388cc91ba4d5eb8d8a370e34d5217" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } health_report { overall_health: HEALTHY } } }
I20260927 00:39:14.355237 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.353810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.0.98.1:39315)
0927 00:39:14.353936 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.353941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.353955 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:14.354048 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.354051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.354114 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.354196 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.354858 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.354861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.354863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.354873 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.354890 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.354898 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.354982 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.355055 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.355099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:39:14.361465 5291 raft_consensus.cc:1261] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 1 FOLLOWER]: Refusing update from remote peer 0ba388cc91ba4d5eb8d8a370e34d5217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:14.361548 5422 raft_consensus.cc:1261] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 1 FOLLOWER]: Refusing update from remote peer 0ba388cc91ba4d5eb8d8a370e34d5217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:14.362239 5491 consensus_queue.cc:1035] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "719b00ea5a854c5e8ecb3ef6b40d3c86" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:14.362488 5495 consensus_queue.cc:1035] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 0ba388cc91ba4d5eb8d8a370e34d5217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b782eae4403940718c25c07c389fb625" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:14.368835 5498 mvcc.cc:205] Tried to move back new op lower bound from 7333763294656438272 to 7333763294589517824. Current Snapshot: MvccSnapshot[applied={T|T < 7333763294656438272}]
I20260927 00:39:14.371310 5499 mvcc.cc:205] Tried to move back new op lower bound from 7333763294656438272 to 7333763294589517824. Current Snapshot: MvccSnapshot[applied={T|T < 7333763294656438272}]
I20260927 00:39:14.372449 5500 mvcc.cc:205] Tried to move back new op lower bound from 7333763294656438272 to 7333763294589517824. Current Snapshot: MvccSnapshot[applied={T|T < 7333763294656438272}]
I20260927 00:39:14.379298 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:14.379487 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:14.381951 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b84488bf1a7f41b7ae653d4a8567d9c3: Sending DeleteTablet for 3 replicas of tablet 5dbf18fb320b4bc59b0cf3adec11a4a7
I20260927 00:39:14.382946 5402 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:14 UTC) from {username='slave'} at 127.0.0.1:42848
I20260927 00:39:14.382947 5271 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:14 UTC) from {username='slave'} at 127.0.0.1:40416
I20260927 00:39:14.382953 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:14 UTC) from {username='slave'} at 127.0.0.1:41520
I20260927 00:39:14.383453 5508 tablet_replica.cc:330] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: stopping tablet replica
I20260927 00:39:14.383865 5508 raft_consensus.cc:2229] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:14.383899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5078
I20260927 00:39:14.385445 5509 tablet_replica.cc:330] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: stopping tablet replica
I20260927 00:39:14.385766 5509 raft_consensus.cc:2229] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:14.385915 5508 raft_consensus.cc:2258] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:14.386051 5509 raft_consensus.cc:2258] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:14.387293 5508 ts_tablet_manager.cc:1905] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:14.387514 5509 ts_tablet_manager.cc:1905] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:14.389678 5508 ts_tablet_manager.cc:1918] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:14.389765 5508 log.cc:1196] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/5dbf18fb320b4bc59b0cf3adec11a4a7
I20260927 00:39:14.390027 5508 ts_tablet_manager.cc:1939] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P 719b00ea5a854c5e8ecb3ef6b40d3c86: Deleting consensus metadata
I20260927 00:39:14.390314 5509 ts_tablet_manager.cc:1918] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:14.390444 5509 log.cc:1196] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/5dbf18fb320b4bc59b0cf3adec11a4a7
I20260927 00:39:14.390722 5509 ts_tablet_manager.cc:1939] T 5dbf18fb320b4bc59b0cf3adec11a4a7 P b782eae4403940718c25c07c389fb625: Deleting consensus metadata
I20260927 00:39:14.391517 5005 catalog_manager.cc:4841] TS 719b00ea5a854c5e8ecb3ef6b40d3c86 (127.0.98.3:43119): tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 (table test-table [id=ae9ed1f8899b4836854aa419efcbac80]) successfully deleted
W20260927 00:39:14.391719 5003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39315 (error 108)
I20260927 00:39:14.391963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5209
I20260927 00:39:14.391949 5004 catalog_manager.cc:4841] TS b782eae4403940718c25c07c389fb625 (127.0.98.2:33161): tablet 5dbf18fb320b4bc59b0cf3adec11a4a7 (table test-table [id=ae9ed1f8899b4836854aa419efcbac80]) successfully deleted
I20260927 00:39:14.393358 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.393070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39315 (local address 127.0.0.1:41536)
0927 00:39:14.393207 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.393281 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:39:14.393510 5003 catalog_manager.cc:4584] TS 0ba388cc91ba4d5eb8d8a370e34d5217 (127.0.98.1:39315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dbf18fb320b4bc59b0cf3adec11a4a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39315: connect: Connection refused (error 111)
I20260927 00:39:14.397111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5340
I20260927 00:39:14.401831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 4987
2026-09-27T00:39:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (978 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:39:14.422077 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:39:14.423605 392 webserver.cc:460] Webserver started at https://127.0.0.1:33679/ using document root <none> and password file <none>
I20260927 00:39:14.423761 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:39:14.424013 392 webserver.cc:460] Webserver started at http://127.0.0.1:42909/ using document root <none> and password file <none>
2026-09-27T00:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:14Z Disabled control of system clock
I20260927 00:39:14.432775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37409
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/testchainca.cert with env {}
W20260927 00:39:14.543124 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:14.543418 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:14.543460 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:14.547613 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:14.547686 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:14.547700 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:14.547716 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:14.547730 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:39:14.547744 5517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:39:14.547756 5517 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33679/jwks.json
I20260927 00:39:14.553316 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:14.554378 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:14.555343 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:14.564532 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.564541 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.564656 5517 server_base.cc:1034] running on GCE node
W20260927 00:39:14.564541 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.565634 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:14.566910 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:14.568173 5517 hybrid_clock.cc:648] HybridClock initialized: now 1790469554568160 us; error 29 us; skew 500 ppm
I20260927 00:39:14.570246 5517 webserver.cc:460] Webserver started at http://127.0.98.62:39021/ using document root <none> and password file <none>
I20260927 00:39:14.570839 5517 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:14.570910 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:14.571110 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:14.573405 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "e3c90a85eaf748e3a970fc57f19512b5"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "b35e02c1e7057735bd1219238c064fdd"
server_key_iv: "87e6f22c0498c09df0ee56d1762e560f"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.574009 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3c90a85eaf748e3a970fc57f19512b5"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "b35e02c1e7057735bd1219238c064fdd"
server_key_iv: "87e6f22c0498c09df0ee56d1762e560f"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.578099 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:14.580825 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.582247 5517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:14.582377 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "e3c90a85eaf748e3a970fc57f19512b5"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "b35e02c1e7057735bd1219238c064fdd"
server_key_iv: "87e6f22c0498c09df0ee56d1762e560f"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.582489 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:14.602797 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:14.603539 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:14.603719 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:14.610435 5517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37409
I20260927 00:39:14.610442 5583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37409 every 8 connection(s)
I20260927 00:39:14.611527 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:14.614974 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:14.618779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5517
I20260927 00:39:14.618909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:14.619202 392 external_mini_cluster.cc:1428] Setting key 997428ebcd2f5d1f97383309a62c65f7
I20260927 00:39:14.621310 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5: Bootstrap starting.
I20260927 00:39:14.624079 5584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:14.625030 5584 log.cc:824] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:14.627255 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5: No bootstrap required, opened a new log
I20260927 00:39:14.630098 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.620722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.62:37409)
0927 00:39:14.621436 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.621458 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.621503 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:39:14.622341 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.622365 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.622915 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.623295 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.623772 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.625297 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.626379 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.627062 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.627209 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:14.627948 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.627955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.627972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.628505 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.628577 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.628604 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.628701 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.628818 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.629124 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260927 00:39:14.631536 5584 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } }
I20260927 00:39:14.631776 5584 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:14.631810 5584 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c90a85eaf748e3a970fc57f19512b5, State: Initialized, Role: FOLLOWER
I20260927 00:39:14.632341 5584 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } }
I20260927 00:39:14.632480 5584 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:14.632552 5584 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:14.632632 5584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:14.632972 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:39:14.633749 5584 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } }
I20260927 00:39:14.634095 5584 leader_election.cc:302] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c90a85eaf748e3a970fc57f19512b5; no voters:
I20260927 00:39:14.634428 5584 leader_election.cc:288] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:14.634524 5589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:14.634738 5589 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [term 1 LEADER]: Becoming Leader. State: Replica: e3c90a85eaf748e3a970fc57f19512b5, State: Running, Role: LEADER
I20260927 00:39:14.635076 5589 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } }
I20260927 00:39:14.635684 5584 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:14.636611 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c90a85eaf748e3a970fc57f19512b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } } }
I20260927 00:39:14.636754 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:14.636967 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c90a85eaf748e3a970fc57f19512b5. Latest consensus state: current_term: 1 leader_uuid: "e3c90a85eaf748e3a970fc57f19512b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c90a85eaf748e3a970fc57f19512b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37409 } } }
I20260927 00:39:14.637058 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:14.637193 5598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:14.639918 5598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:14.646272 5598 catalog_manager.cc:1347] Generated new cluster ID: eb6037c6da474d729ddbd4a91163b8ff
I20260927 00:39:14.646353 5598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:14.662792 5598 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:14.664186 5598 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:14.674870 5598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c90a85eaf748e3a970fc57f19512b5: Generated new TSK 0
I20260927 00:39:14.675668 5598 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:39:14.713626 5614 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:39:14.713728 5614 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:39:14.714406 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.696639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.98.62:37409)
0927 00:39:14.696779 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.696784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.696927 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:39:14.697124 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.697127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.697191 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.697309 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:39:14.697515 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.698435 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.699305 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.699673 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.699777 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:14.700163 (+ 386us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:39:14.713950 (+ 13787us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:39:14.714255 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:39:14.714493 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:39:14.715720 5613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:14.696527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37409 (local address 127.0.0.1:60542)
0927 00:39:14.696845 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:14.696858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:14.696973 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:14.697311 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:14.697313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:14.697319 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:39:14.697421 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:14.697426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.698558 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.698561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:14.699198 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:14.699206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.699800 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.699826 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:14.699900 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:14.699954 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:39:14.714282 (+ 14328us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:39:14.714522 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:37409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33679/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:39:14.716482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5517
2026-09-27T00:39:14Z chronyd exiting
I20260927 00:39:14.753561 392 test_util.cc:175] -----------------------------------------------
I20260927 00:39:14.753659 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469528995939-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:39:14.754735 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:39:14.755581 392 webserver.cc:460] Webserver started at https://127.0.0.1:35239/ using document root <none> and password file <none>
I20260927 00:39:14.755689 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:39:14.755930 392 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
2026-09-27T00:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:14Z Disabled control of system clock
I20260927 00:39:14.764346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43801
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35239/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/testchainca.cert with env {}
W20260927 00:39:14.877179 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:14.877481 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:14.877521 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:14.881660 5620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:14.881734 5620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:14.881749 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:14.881765 5620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:14.881779 5620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:39:14.881795 5620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:39:14.881810 5620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35239/jwks.json
I20260927 00:39:14.887550 5620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35239/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:14.888717 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:14.889758 5620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:14.898849 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:14.898823 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.898912 5620 server_base.cc:1034] running on GCE node
W20260927 00:39:14.898828 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:14.899669 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:14.900954 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:14.902200 5620 hybrid_clock.cc:648] HybridClock initialized: now 1790469554902167 us; error 47 us; skew 500 ppm
I20260927 00:39:14.904227 5620 webserver.cc:460] Webserver started at http://127.0.98.62:36371/ using document root <none> and password file <none>
I20260927 00:39:14.904860 5620 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:14.904930 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:14.905130 5620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:14.907246 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "e73831689c8342a9b3535c6adb9bc75a"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "0305b6bdb12c2dad0d11ffc8874f6f13"
server_key_iv: "da89f840a0783135b1f5afd27dd17145"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.907799 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "e73831689c8342a9b3535c6adb9bc75a"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "0305b6bdb12c2dad0d11ffc8874f6f13"
server_key_iv: "da89f840a0783135b1f5afd27dd17145"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.911749 5620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:39:14.914652 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:14.916067 5620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:39:14.916275 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "e73831689c8342a9b3535c6adb9bc75a"
format_stamp: "Formatted at 2026-09-27 00:39:14 on dist-test-slave-xltq"
server_key: "0305b6bdb12c2dad0d11ffc8874f6f13"
server_key_iv: "da89f840a0783135b1f5afd27dd17145"
server_key_version: "encryptionkey@0"
I20260927 00:39:14.916397 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:14.932435 5620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:14.933151 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:14.933327 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:14.939860 5620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43801
I20260927 00:39:14.939891 5686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43801 every 8 connection(s)
I20260927 00:39:14.940816 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:14.944097 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:14.949887 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: Bootstrap starting.
I20260927 00:39:14.950687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5620
I20260927 00:39:14.950809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:14.951105 392 external_mini_cluster.cc:1428] Setting key 292f9c979b060787273bd5e2ad654539
I20260927 00:39:14.952994 5687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:14.953766 5687 log.cc:824] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:14.956173 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: No bootstrap required, opened a new log
I20260927 00:39:14.960434 5687 raft_consensus.cc:348] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260927 00:39:14.960719 5687 raft_consensus.cc:374] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:14.960779 5687 raft_consensus.cc:729] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73831689c8342a9b3535c6adb9bc75a, State: Initialized, Role: FOLLOWER
I20260927 00:39:14.961256 5687 consensus_queue.cc:260] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260927 00:39:14.961366 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:14.952412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.62:43801)
0927 00:39:14.953214 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:14.953230 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:14.953270 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:39:14.953985 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:14.954006 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:14.954527 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:14.954908 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:14.955345 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.956514 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.957450 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:14.957982 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:14.958148 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:14.959250 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:14.959256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:14.959273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:14.959745 (+ 472us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:14.959811 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:14.959834 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:14.959931 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:14.960093 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:39:14.960432 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":131,"threads_started":1}
I20260927 00:39:14.961407 5687 raft_consensus.cc:388] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:14.961477 5687 raft_consensus.cc:482] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:14.961578 5687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:14.962803 5687 raft_consensus.cc:504] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260927 00:39:14.963155 5687 leader_election.cc:302] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e73831689c8342a9b3535c6adb9bc75a; no voters:
I20260927 00:39:14.963481 5687 leader_election.cc:288] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:14.963626 5692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:14.963865 5692 raft_consensus.cc:686] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [term 1 LEADER]: Becoming Leader. State: Replica: e73831689c8342a9b3535c6adb9bc75a, State: Running, Role: LEADER
I20260927 00:39:14.964287 5692 consensus_queue.cc:237] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } }
I20260927 00:39:14.964694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:39:14.964905 5687 sys_catalog.cc:564] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:14.966105 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e73831689c8342a9b3535c6adb9bc75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } } }
I20260927 00:39:14.966238 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:14.966540 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73831689c8342a9b3535c6adb9bc75a. Latest consensus state: current_term: 1 leader_uuid: "e73831689c8342a9b3535c6adb9bc75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73831689c8342a9b3535c6adb9bc75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43801 } } }
I20260927 00:39:14.966632 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a [sys.catalog]: This master's current role is: LEADER
W20260927 00:39:14.969653 5707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:39:14.969741 5707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:39:14.969854 5701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:14.970913 5701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:14.977303 5701 catalog_manager.cc:1347] Generated new cluster ID: a1e5c22da3f84a818f2d95b781ca93d9
I20260927 00:39:14.977384 5701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:15.000861 5701 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:15.002252 5701 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:15.012826 5701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e73831689c8342a9b3535c6adb9bc75a: Generated new TSK 0
I20260927 00:39:15.013738 5701 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:39:15.041256 5717 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:39:15.041370 5717 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:39:15.041993 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:15.024843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.0.98.62:43801)
0927 00:39:15.024961 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:15.024964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:15.025111 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:39:15.025279 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:15.025283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:15.025345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:15.025477 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:39:15.025761 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.026634 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.027545 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.027903 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.028010 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.028300 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:39:15.041536 (+ 13236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:39:15.041847 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 00:39:15.042079 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:39:15.042100 5716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:15.024748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43801 (local address 127.0.0.1:34712)
0927 00:39:15.025018 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:15.025032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:15.025134 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:15.025527 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:15.025530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:15.025538 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:39:15.025646 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.025651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.026785 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.026788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.027434 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.027442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.027992 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.028012 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.028072 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.028095 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:39:15.041764 (+ 13669us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:39:15.041935 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260927 00:39:15.042755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5620
2026-09-27T00:39:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:39:15.081809 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:39:15.082666 392 webserver.cc:460] Webserver started at https://127.0.0.1:33995/ using document root <none> and password file <none>
I20260927 00:39:15.082769 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:39:15.083011 392 webserver.cc:460] Webserver started at http://127.0.0.1:34513/ using document root <none> and password file <none>
2026-09-27T00:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:15Z Disabled control of system clock
I20260927 00:39:15.091409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36129
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33995/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:39:15.203261 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:15.203548 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:15.203588 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:15.207794 5723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:15.207880 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:15.207894 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:15.207911 5723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:15.207926 5723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:39:15.207940 5723 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:39:15.207957 5723 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33995/jwks.json
I20260927 00:39:15.212927 5723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33995/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:15.214078 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:15.215463 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:15.224308 5729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:15.224440 5723 server_base.cc:1034] running on GCE node
W20260927 00:39:15.224514 5728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:15.224562 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:15.225292 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:15.226456 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:15.227631 5723 hybrid_clock.cc:648] HybridClock initialized: now 1790469555227600 us; error 44 us; skew 500 ppm
I20260927 00:39:15.229756 5723 webserver.cc:460] Webserver started at http://127.0.98.62:35189/ using document root <none> and password file <none>
I20260927 00:39:15.230360 5723 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:15.230433 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:15.230630 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:15.232903 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "99661e8f7bea447fa70fc36e6f99339e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "81a28b609c60f0976a0b572c938c1aca"
server_key_iv: "9f5c27b186d661d58e009dada503b756"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.233493 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "99661e8f7bea447fa70fc36e6f99339e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "81a28b609c60f0976a0b572c938c1aca"
server_key_iv: "9f5c27b186d661d58e009dada503b756"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.237452 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:15.240063 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:15.241458 5723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:15.241586 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "99661e8f7bea447fa70fc36e6f99339e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "81a28b609c60f0976a0b572c938c1aca"
server_key_iv: "9f5c27b186d661d58e009dada503b756"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.241696 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:15.254789 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:15.255517 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:15.255689 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:15.262652 5723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36129
I20260927 00:39:15.262677 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36129 every 8 connection(s)
I20260927 00:39:15.263614 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:15.267047 5790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:15.267405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5723
I20260927 00:39:15.267513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:15.267763 392 external_mini_cluster.cc:1428] Setting key ab88a14ab64adabd40217d06b9a630e0
I20260927 00:39:15.273278 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e: Bootstrap starting.
I20260927 00:39:15.275988 5790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:15.276893 5790 log.cc:824] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:15.277837 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:15.269096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.0.98.62:36129)
0927 00:39:15.269749 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:15.269764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:15.269801 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:39:15.270490 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:15.270507 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:15.270940 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:15.271240 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:15.271603 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.272664 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.273965 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.274485 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.274616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.275613 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:15.275619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:15.275632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:15.276099 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:15.276186 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:15.276245 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:15.276342 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:15.276530 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 00:39:15.276823 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260927 00:39:15.278985 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e: No bootstrap required, opened a new log
I20260927 00:39:15.280769 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:39:15.282975 5790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } }
I20260927 00:39:15.283228 5790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:15.283272 5790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99661e8f7bea447fa70fc36e6f99339e, State: Initialized, Role: FOLLOWER
I20260927 00:39:15.283689 5790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } }
I20260927 00:39:15.283819 5790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:15.283877 5790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:15.283972 5790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:15.284970 5790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } }
I20260927 00:39:15.285329 5790 leader_election.cc:302] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99661e8f7bea447fa70fc36e6f99339e; no voters:
I20260927 00:39:15.285627 5790 leader_election.cc:288] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:15.285791 5795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:15.286027 5795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [term 1 LEADER]: Becoming Leader. State: Replica: 99661e8f7bea447fa70fc36e6f99339e, State: Running, Role: LEADER
I20260927 00:39:15.286326 5795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } }
I20260927 00:39:15.286836 5790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:15.287953 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99661e8f7bea447fa70fc36e6f99339e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } } }
I20260927 00:39:15.288077 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:15.288345 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99661e8f7bea447fa70fc36e6f99339e. Latest consensus state: current_term: 1 leader_uuid: "99661e8f7bea447fa70fc36e6f99339e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99661e8f7bea447fa70fc36e6f99339e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36129 } } }
I20260927 00:39:15.288434 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:15.288831 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:15.291613 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:15.297559 5809 catalog_manager.cc:1347] Generated new cluster ID: a122beebbe8e4204988a82160162b3af
I20260927 00:39:15.297658 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:15.311172 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:15.312224 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:15.323590 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99661e8f7bea447fa70fc36e6f99339e: Generated new TSK 0
I20260927 00:39:15.324368 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:15.341722 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:15.336066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.0.98.62:36129)
0927 00:39:15.336233 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:15.336237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:15.336497 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:39:15.336695 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:15.336698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:15.336751 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:15.336849 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:39:15.337170 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.337763 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.339062 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.339363 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.339457 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.339826 (+ 369us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:39:15.341328 (+ 1502us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33995/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:39:15.341576 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33995/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:39:15.341797 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33995/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 00:39:15.343698 5819 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:15.335974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36129 (local address 127.0.0.1:54464)
0927 00:39:15.336382 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:15.336397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:15.336522 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:15.336862 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:15.336865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:15.336872 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:39:15.336982 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.336988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.337877 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.337880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.338897 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.338909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.339466 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.339494 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.339584 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.339613 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:39:15.341525 (+ 1912us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33995/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:39:15.343392 (+ 1867us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36129: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33995/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":1525,"thread_start_us":230,"threads_started":1}
I20260927 00:39:15.344375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5723
E20260927 00:39:15.352625 5820 webserver.cc:533] Webserver: sslize failed:
E20260927 00:39:15.352711 5820 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:39:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (301 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:15Z Disabled control of system clock
I20260927 00:39:15.391750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33013
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33013
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/testchainca.cert with env {}
W20260927 00:39:15.505839 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:15.506146 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:15.506206 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:15.510452 5824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:15.510536 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:15.510560 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:15.510581 5824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:15.510601 5824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:39:15.510630 5824 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:39:15.510653 5824 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:39:15.516470 5824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:15.517606 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:15.518690 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:15.527740 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:15.527693 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:15.527705 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:15.527985 5824 server_base.cc:1034] running on GCE node
I20260927 00:39:15.528695 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:15.529917 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:15.531097 5824 hybrid_clock.cc:648] HybridClock initialized: now 1790469555531075 us; error 39 us; skew 500 ppm
I20260927 00:39:15.533309 5824 webserver.cc:460] Webserver started at http://127.0.98.62:38479/ using document root <none> and password file <none>
I20260927 00:39:15.533924 5824 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:15.533995 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:15.534194 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:15.536453 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "6469fbdfb08842e9bf3533e2b9c58e6e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "0e3f5e37edb87e4da743332c513833b0"
server_key_iv: "13f8ca384742f6ea6bd53c3b2a92959f"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.537041 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "6469fbdfb08842e9bf3533e2b9c58e6e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "0e3f5e37edb87e4da743332c513833b0"
server_key_iv: "13f8ca384742f6ea6bd53c3b2a92959f"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.541049 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:15.543938 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:15.545492 5824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:15.545637 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "6469fbdfb08842e9bf3533e2b9c58e6e"
format_stamp: "Formatted at 2026-09-27 00:39:15 on dist-test-slave-xltq"
server_key: "0e3f5e37edb87e4da743332c513833b0"
server_key_iv: "13f8ca384742f6ea6bd53c3b2a92959f"
server_key_version: "encryptionkey@0"
I20260927 00:39:15.545756 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:15.562721 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:15.563475 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:15.563647 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:15.570855 5824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33013
I20260927 00:39:15.570884 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33013 every 8 connection(s)
I20260927 00:39:15.571856 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:15.575548 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:15.577939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5824
I20260927 00:39:15.578068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:15.578436 392 external_mini_cluster.cc:1428] Setting key 2415741dc79254678d6919067b12199a
I20260927 00:39:15.582727 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: Bootstrap starting.
I20260927 00:39:15.585602 5891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:15.586553 5891 log.cc:824] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:15.588886 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: No bootstrap required, opened a new log
I20260927 00:39:15.589679 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:15.580044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.0.98.62:33013)
0927 00:39:15.580830 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:15.580849 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:15.580893 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:39:15.581731 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:15.581757 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:15.582304 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:15.582742 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:15.583268 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.584767 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.585788 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.586484 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.586663 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.587500 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:15.587506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:15.587524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:15.588020 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:15.588093 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:15.588119 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:15.588240 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:15.588420 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 00:39:15.588716 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":163,"threads_started":1}
I20260927 00:39:15.592808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:39:15.592783 5891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } }
I20260927 00:39:15.593082 5891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:15.593139 5891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6469fbdfb08842e9bf3533e2b9c58e6e, State: Initialized, Role: FOLLOWER
I20260927 00:39:15.593668 5891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } }
I20260927 00:39:15.593835 5891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:15.593907 5891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:15.594020 5891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:15.595175 5891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } }
I20260927 00:39:15.595526 5891 leader_election.cc:302] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6469fbdfb08842e9bf3533e2b9c58e6e; no voters:
I20260927 00:39:15.595835 5891 leader_election.cc:288] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:15.595995 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:15.596235 5896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [term 1 LEADER]: Becoming Leader. State: Replica: 6469fbdfb08842e9bf3533e2b9c58e6e, State: Running, Role: LEADER
I20260927 00:39:15.596704 5896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } }
I20260927 00:39:15.597194 5891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:15.597892 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6469fbdfb08842e9bf3533e2b9c58e6e. Latest consensus state: current_term: 1 leader_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } }
I20260927 00:39:15.598075 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:15.598294 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6469fbdfb08842e9bf3533e2b9c58e6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33013 } } }
I20260927 00:39:15.598382 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e [sys.catalog]: This master's current role is: LEADER
W20260927 00:39:15.602264 5911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:39:15.602367 5911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:39:15.602473 5905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:15.603291 5905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:15.609253 5905 catalog_manager.cc:1347] Generated new cluster ID: 56a9ea6da11742e1befcf4e77af0e5fe
I20260927 00:39:15.609342 5905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:15.633626 5905 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:15.634975 5905 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:15.641247 5905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6469fbdfb08842e9bf3533e2b9c58e6e: Generated new TSK 0
I20260927 00:39:15.642006 5905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:15.749416 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:15.650313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.98.62:33013)
0927 00:39:15.650462 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:15.650467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:15.650801 (+ 334us) server_negotiation.cc:407] Connection header received
0927 00:39:15.650968 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:15.650972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:15.651036 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:15.651163 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:39:15.651384 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.652321 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.653657 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.653900 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.653990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.654351 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:39:15.748930 (+ 94579us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:39:15.749196 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 00:39:15.749627 5920 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:15.650323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33013 (local address 127.0.0.1:57204)
0927 00:39:15.650701 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:15.650717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:15.650841 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:15.651172 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:15.651175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:15.651184 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:39:15.651287 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.651292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.652458 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.652462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.653489 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:15.653500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:15.654003 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:15.654031 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:15.654123 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:15.654154 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:39:15.749186 (+ 95032us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:39:15.749421 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":118,"threads_started":1}
W20260927 00:39:15.749657 5894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260927 00:39:15.750375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5824
2026-09-27T00:39:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (385 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (129 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:16 dist-test-slave-xltq krb5kdc[5932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:16 dist-test-slave-xltq krb5kdc[5932](info): set up 2 sockets
Sep 27 00:39:16 dist-test-slave-xltq krb5kdc[5932](info): commencing operation
krb5kdc: starting...
W20260927 00:39:16.984332 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:17.108726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:17.229671 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:17.230039 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:17.230098 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:17.234826 5945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:17.234905 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:17.234925 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:17.234941 5945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:17.234956 5945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:17.241032 5945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:17.242210 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:17.243321 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:17.253037 5945 server_base.cc:1034] running on GCE node
W20260927 00:39:17.252993 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:17.252988 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:17.252988 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:17.254033 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:17.255267 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:17.256443 5945 hybrid_clock.cc:648] HybridClock initialized: now 1790469557256410 us; error 49 us; skew 500 ppm
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:17.259413 5945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:17.260658 5945 webserver.cc:460] Webserver started at http://127.0.98.62:46453/ using document root <none> and password file <none>
I20260927 00:39:17.261230 5945 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:17.261307 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:17.261521 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:17.263732 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "981f319558c94101a748b88c9a23ca61"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "6cfbea2aad13a43000b271cccf22d0e3"
server_key_iv: "082f440e751e06453aa1b8029eb235b6"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.264384 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "981f319558c94101a748b88c9a23ca61"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "6cfbea2aad13a43000b271cccf22d0e3"
server_key_iv: "082f440e751e06453aa1b8029eb235b6"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.268605 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:17.271344 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.272872 5945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:17.273048 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "981f319558c94101a748b88c9a23ca61"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "6cfbea2aad13a43000b271cccf22d0e3"
server_key_iv: "082f440e751e06453aa1b8029eb235b6"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.273183 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:17.286710 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:17.290277 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:17.290474 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:17.297657 5945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32801
I20260927 00:39:17.297904 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32801 every 8 connection(s)
I20260927 00:39:17.298656 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:17.301614 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:17.305526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 5945
I20260927 00:39:17.305629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:17.305878 392 external_mini_cluster.cc:1428] Setting key 46d1c00087398e1a2a985be6e508fac9
I20260927 00:39:17.307744 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61: Bootstrap starting.
I20260927 00:39:17.310329 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:17.311084 6013 log.cc:824] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:17.313301 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61: No bootstrap required, opened a new log
I20260927 00:39:17.316932 6013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } }
I20260927 00:39:17.317219 6013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:17.317286 6013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981f319558c94101a748b88c9a23ca61, State: Initialized, Role: FOLLOWER
I20260927 00:39:17.317739 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } }
I20260927 00:39:17.317874 6013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:17.317942 6013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:17.318042 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:17.324563 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:17.307503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.98.62:32801)
0927 00:39:17.307932 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:17.307984 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:17.308022 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:39:17.308722 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:17.308741 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:17.309097 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:17.309403 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:17.310331 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.311494 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.312445 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.313164 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.313326 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:17.319446 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:17.319453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:17.319470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:17.319507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:17.321541 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.322024 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.322030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.322035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.322150 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.322356 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.322358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.322359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.322851 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:17.323013 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:17.323287 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:39:17.323651 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260927 00:39:17.370834 6013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } }
I20260927 00:39:17.371377 6013 leader_election.cc:302] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981f319558c94101a748b88c9a23ca61; no voters:
I20260927 00:39:17.371762 6013 leader_election.cc:288] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:17.371902 6018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:17.372134 6018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [term 1 LEADER]: Becoming Leader. State: Replica: 981f319558c94101a748b88c9a23ca61, State: Running, Role: LEADER
I20260927 00:39:17.372605 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } }
I20260927 00:39:17.373191 6013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:17.374588 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981f319558c94101a748b88c9a23ca61. Latest consensus state: current_term: 1 leader_uuid: "981f319558c94101a748b88c9a23ca61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } } }
I20260927 00:39:17.374676 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981f319558c94101a748b88c9a23ca61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981f319558c94101a748b88c9a23ca61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32801 } } }
I20260927 00:39:17.374737 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:17.374737 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:17.375176 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:17.378379 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:17.384449 6024 catalog_manager.cc:1347] Generated new cluster ID: aebdd5cb20224b50a4fb29ec3eb8a6a0
I20260927 00:39:17.384534 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:17.399089 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:17.400193 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:17.406045 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981f319558c94101a748b88c9a23ca61: Generated new TSK 0
I20260927 00:39:17.406826 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:17.493741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:17.606052 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:17.606334 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:17.606379 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:17.610598 6039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:17.610666 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:17.610726 6039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:17.616297 6039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:17.617461 6039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:17.618594 6039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:17.628448 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:17.628448 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:17.629029 6039 server_base.cc:1034] running on GCE node
W20260927 00:39:17.628414 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:17.629572 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:17.630240 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:17.631417 6039 hybrid_clock.cc:648] HybridClock initialized: now 1790469557631389 us; error 38 us; skew 500 ppm
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:17.634224 6039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:17.635138 6039 webserver.cc:460] Webserver started at http://127.0.98.1:39421/ using document root <none> and password file <none>
I20260927 00:39:17.635689 6039 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:17.635749 6039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:17.635937 6039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:17.638110 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a4aad41fba446fbbfad034ddbf040b1"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "bb1ec6a259419518a80a3be6397caeb1"
server_key_iv: "a7a55fe058b9c235d3f793231778806b"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.638645 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a4aad41fba446fbbfad034ddbf040b1"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "bb1ec6a259419518a80a3be6397caeb1"
server_key_iv: "a7a55fe058b9c235d3f793231778806b"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.642510 6039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:17.644981 6054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.646224 6039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:39:17.646327 6039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "8a4aad41fba446fbbfad034ddbf040b1"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "bb1ec6a259419518a80a3be6397caeb1"
server_key_iv: "a7a55fe058b9c235d3f793231778806b"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.646435 6039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:17.678566 6039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:17.681988 6039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:17.682214 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:17.682825 6039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:17.683667 6039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:17.683710 6039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.683755 6039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:17.683784 6039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.692612 6039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34903
I20260927 00:39:17.692700 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34903 every 8 connection(s)
I20260927 00:39:17.693538 6039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:17.700507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6039
I20260927 00:39:17.700651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:17.700943 392 external_mini_cluster.cc:1428] Setting key 9134ec88736bbf32822011cc1356849b
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:17.708484 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:17.695327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57643 (local address 127.0.98.62:32801)
0927 00:39:17.695456 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:17.695542 (+ 86us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:17.696336 (+ 794us) server_negotiation.cc:407] Connection header received
0927 00:39:17.697900 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:17.697903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:17.697973 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:17.698047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:17.699383 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.699922 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.701531 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.701762 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.701873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:17.704801 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:17.704804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:17.704806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:17.704833 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:17.706260 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.706840 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.706843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.706844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.706941 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.707288 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.707290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.707292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.707489 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:17.707583 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:17.708150 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 00:39:17.708309 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:39:17.709347 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:17.695688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32801 (local address 127.0.98.1:57643)
0927 00:39:17.696147 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:17.696229 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:17.697650 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:17.698254 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:17.698261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:17.698720 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:17.699229 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:17.699243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.700053 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.700056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:17.701402 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:17.701413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.701909 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.701913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:17.702101 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:17.702729 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:17.702750 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:17.704586 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:17.706411 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:17.706418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:17.706436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:17.706728 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:17.707040 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:17.707043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:17.707045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:17.707193 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:17.707588 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:17.707596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:17.707909 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 00:39:17.708080 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:39:17.708463 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:17.710682 6168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32801
I20260927 00:39:17.710925 6168 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:17.711490 6168 heartbeater.cc:511] Master 127.0.98.62:32801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:17.713140 5977 ts_manager.cc:194] Registered new tserver with Master: 8a4aad41fba446fbbfad034ddbf040b1 (127.0.98.1:34903)
I20260927 00:39:17.714287 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:17.729910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:17.846042 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:17.846334 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:17.846379 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:17.850576 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:17.850666 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:17.850739 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:17.856379 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:17.857438 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:17.858441 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:17.868425 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:17.868425 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:17.868463 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:17.868443 6173 server_base.cc:1034] running on GCE node
I20260927 00:39:17.869123 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:17.869782 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:17.870950 6173 hybrid_clock.cc:648] HybridClock initialized: now 1790469557870927 us; error 39 us; skew 500 ppm
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:17.873790 6173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:17.874778 6173 webserver.cc:460] Webserver started at http://127.0.98.2:46275/ using document root <none> and password file <none>
I20260927 00:39:17.875407 6173 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:17.875470 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:17.875662 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:17.877981 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "a72b0edf595e4f58b050aa3426bb9bfc"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "54b1c7365f8bc516ff34b96c7bc7983b"
server_key_iv: "e8b913631e2e900fe64def3971bf6a0a"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.878501 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a72b0edf595e4f58b050aa3426bb9bfc"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "54b1c7365f8bc516ff34b96c7bc7983b"
server_key_iv: "e8b913631e2e900fe64def3971bf6a0a"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.882256 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 00:39:17.884863 6188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.886291 6173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:17.886426 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "a72b0edf595e4f58b050aa3426bb9bfc"
format_stamp: "Formatted at 2026-09-27 00:39:17 on dist-test-slave-xltq"
server_key: "54b1c7365f8bc516ff34b96c7bc7983b"
server_key_iv: "e8b913631e2e900fe64def3971bf6a0a"
server_key_version: "encryptionkey@0"
I20260927 00:39:17.886565 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:17.896986 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:17.900352 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:17.900542 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:17.901134 6173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:17.901947 6173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:17.901993 6173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.902046 6173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:17.902074 6173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:17.910578 6173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44779
I20260927 00:39:17.910650 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44779 every 8 connection(s)
I20260927 00:39:17.911487 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:17.916540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6173
I20260927 00:39:17.916653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:17.916898 392 external_mini_cluster.cc:1428] Setting key 7e9bed1c75a1ef3cd51e934651edb211
Sep 27 00:39:17 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:17.926229 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:17.913371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50231 (local address 127.0.98.62:32801)
0927 00:39:17.913576 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:17.913579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:17.914328 (+ 749us) server_negotiation.cc:407] Connection header received
0927 00:39:17.915696 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:17.915699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:17.915746 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:17.915830 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:17.917210 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.917836 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.919149 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.919454 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.919588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:17.922437 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:17.922440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:17.922442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:17.922467 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:17.923862 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.924523 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.924526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.924527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.924612 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:17.925009 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:17.925012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:17.925013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:17.925224 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:17.925300 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:17.925969 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 00:39:17.926094 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 00:39:17.927141 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:17.913753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32801 (local address 127.0.98.2:50231)
0927 00:39:17.914181 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:17.914226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:17.915479 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:17.916021 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:17.916028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:17.916489 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:17.917064 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:17.917081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.918005 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.918009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:17.919028 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:17.919039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:17.919648 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:17.919653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:17.919875 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:17.920557 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:17.920578 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:17.922238 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:17.923983 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:17.923990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:17.924016 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:17.924422 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:17.924704 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:17.924707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:17.924709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:17.924894 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:17.925324 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:17.925331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:17.925688 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 00:39:17.925901 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:39:17.926280 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260927 00:39:17.928311 6302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32801
I20260927 00:39:17.928612 6302 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:17.929184 6302 heartbeater.cc:511] Master 127.0.98.62:32801 requested a full tablet report, sending...
I20260927 00:39:17.930330 5977 ts_manager.cc:194] Registered new tserver with Master: a72b0edf595e4f58b050aa3426bb9bfc (127.0.98.2:44779)
I20260927 00:39:17.931005 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:17.945631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:18.058439 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:18.058738 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:18.058784 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:18.063021 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:18.063087 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:18.063148 6307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:18.068625 6307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:18.069677 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:18.070669 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:18.081073 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:18.081230 6307 server_base.cc:1034] running on GCE node
W20260927 00:39:18.081076 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:18.081073 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:18.081933 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:18.082612 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:18.083789 6307 hybrid_clock.cc:648] HybridClock initialized: now 1790469558083770 us; error 33 us; skew 500 ppm
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.086611 6307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:18.087549 6307 webserver.cc:460] Webserver started at http://127.0.98.3:41781/ using document root <none> and password file <none>
I20260927 00:39:18.088140 6307 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:18.088241 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:18.088454 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:18.090644 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "a37c90db0b504713b4ef00cc32a8501a"
format_stamp: "Formatted at 2026-09-27 00:39:18 on dist-test-slave-xltq"
server_key: "7ca5cf055a37d7cbae07a243fda08354"
server_key_iv: "bd9b492fd70dbcd6987bcf05a7eff94b"
server_key_version: "encryptionkey@0"
I20260927 00:39:18.091269 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a37c90db0b504713b4ef00cc32a8501a"
format_stamp: "Formatted at 2026-09-27 00:39:18 on dist-test-slave-xltq"
server_key: "7ca5cf055a37d7cbae07a243fda08354"
server_key_iv: "bd9b492fd70dbcd6987bcf05a7eff94b"
server_key_version: "encryptionkey@0"
I20260927 00:39:18.095270 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:18.097898 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:18.099478 6307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:18.099609 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "a37c90db0b504713b4ef00cc32a8501a"
format_stamp: "Formatted at 2026-09-27 00:39:18 on dist-test-slave-xltq"
server_key: "7ca5cf055a37d7cbae07a243fda08354"
server_key_iv: "bd9b492fd70dbcd6987bcf05a7eff94b"
server_key_version: "encryptionkey@0"
I20260927 00:39:18.099736 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:18.109908 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:18.113299 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:18.113482 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:18.114074 6307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:18.114905 6307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:18.114956 6307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:18.114996 6307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:18.115025 6307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:18.123876 6307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35333
I20260927 00:39:18.123894 6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35333 every 8 connection(s)
I20260927 00:39:18.124876 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:18.132085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6307
I20260927 00:39:18.132237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:18.132498 392 external_mini_cluster.cc:1428] Setting key 568fe52f701dfde1842d8869d78aa97e
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:18.139392 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:18.126916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44701 (local address 127.0.98.62:32801)
0927 00:39:18.127051 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.127056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.128099 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 00:39:18.129597 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.129601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.129661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.129735 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.131062 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.131585 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.132924 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.133163 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.133259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:18.135863 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:18.135866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:18.135868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:18.135895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:18.137378 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.137915 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.137918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.137919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.137988 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.138359 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.138361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.138362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.138543 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:18.138627 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:18.139151 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 00:39:18.139267 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:39:18.140425 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:18.127380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32801 (local address 127.0.98.3:44701)
0927 00:39:18.127880 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:18.127928 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:18.129351 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:18.129951 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:18.129960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:18.130366 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:18.130895 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:18.130910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.131715 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.131718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:18.132796 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:18.132807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.133321 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.133325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:18.133527 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:18.134156 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:18.134173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:18.135692 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:18.137511 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:18.137516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:18.137532 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:18.137815 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:18.138099 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:18.138102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:18.138104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:18.138268 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:18.138626 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:18.138632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:18.138912 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 00:39:18.139089 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:39:18.139390 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260927 00:39:18.141662 6436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32801
I20260927 00:39:18.141881 6436 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:18.142356 6436 heartbeater.cc:511] Master 127.0.98.62:32801 requested a full tablet report, sending...
I20260927 00:39:18.143342 5977 ts_manager.cc:194] Registered new tserver with Master: a37c90db0b504713b4ef00cc32a8501a (127.0.98.3:35333)
I20260927 00:39:18.143898 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44701
I20260927 00:39:18.146368 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/test-admin.keytab.
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.161429 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:39:18.164242 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:18.162880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.0.98.1:34903)
0927 00:39:18.163194 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.163209 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.163232 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:39:18.163280 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.163285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.163469 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.163591 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.163991 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46136: BlockingRecv error: recv got EOF from 127.0.0.1:46136 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
W20260927 00:39:18.164345 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.165370 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.166365 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:39:18.176294 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:18.167309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.0.98.1:34903)
0927 00:39:18.167499 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.167504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.167519 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:18.167560 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.167562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.167602 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.167670 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.168249 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.169144 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.169971 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.170364 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.170463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:18.172716 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:18.172726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:18.172742 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:18.172768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:18.174477 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.174907 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.174912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.174914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.175005 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.175255 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.175258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.175259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.175517 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:18.175712 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:18.175902 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:39:18.176078 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:39:18.180354 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:18.179284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.0.98.1:34903)
0927 00:39:18.179408 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.179412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.179486 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:39:18.179632 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.179635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.179678 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.179739 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.180244 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46152: BlockingRecv error: recv got EOF from 127.0.0.1:46152 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:39:18.180441 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.181418 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.182278 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:39:18.190918 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:18.183227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.0.98.1:34903)
0927 00:39:18.183338 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.183343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.183413 (+ 70us) server_negotiation.cc:407] Connection header received
0927 00:39:18.183592 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.183595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.183636 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.183700 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.184242 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.184763 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.185632 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.185869 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.185959 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:18.188172 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:18.188175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:18.188177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:18.188227 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:18.189369 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.189769 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.189772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.189773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.189834 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.190112 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.190116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.190118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.190335 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:18.190533 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:18.190658 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 00:39:18.190757 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 00:39:18.193589 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:18.192629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.98.1:34903)
0927 00:39:18.192778 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.192782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.192849 (+ 67us) server_negotiation.cc:407] Connection header received
0927 00:39:18.192969 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.192972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.193017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.193076 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.193475 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46168: BlockingRecv error: recv got EOF from 127.0.0.1:46168 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:39:18.193708 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469528995939-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.194861 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:18.195705 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:39:18.204458 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:18.196694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.0.98.1:34903)
0927 00:39:18.196814 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:18.196819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:18.196887 (+ 68us) server_negotiation.cc:407] Connection header received
0927 00:39:18.197006 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:18.197009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:18.197051 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:18.197124 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:18.197612 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.198116 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.198984 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:18.199203 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:18.199288 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:18.201495 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:18.201498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:18.201500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:18.201524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:18.202860 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.203239 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.203243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.203244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.203317 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:18.203582 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:18.203585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:18.203587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:18.203808 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:18.203943 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:18.204152 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 00:39:18.204298 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:39:18.205482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6039
I20260927 00:39:18.210696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6173
I20260927 00:39:18.216411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6307
I20260927 00:39:18.221879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 5945
2026-09-27T00:39:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2215 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[6456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[6456](info): set up 2 sockets
Sep 27 00:39:18 dist-test-slave-xltq krb5kdc[6456](info): commencing operation
krb5kdc: starting...
W20260927 00:39:19.180908 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:39:19.300005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:19.411358 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:19.411664 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:19.411721 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:19.415912 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:19.415989 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:19.416016 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:19.416038 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:19.416059 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:19.421568 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:19.422703 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:19.423753 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:19.432432 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.432418 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.432420 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:19.432525 6469 server_base.cc:1034] running on GCE node
I20260927 00:39:19.433264 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:19.434407 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:19.435560 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790469559435545 us; error 32 us; skew 500 ppm
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:19.442068 6469 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:39:19.443082 6469 webserver.cc:460] Webserver started at http://127.0.98.62:46447/ using document root <none> and password file <none>
I20260927 00:39:19.443661 6469 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:19.443728 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:19.443938 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:19.446165 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "25135c2052484e7d8704e3ebc7cc8f6c"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "483d2366001781f1edce13b013b6771d"
server_key_iv: "7c62fabceaf8b7851cda6e64d86d120d"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.446743 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "25135c2052484e7d8704e3ebc7cc8f6c"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "483d2366001781f1edce13b013b6771d"
server_key_iv: "7c62fabceaf8b7851cda6e64d86d120d"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.450605 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:19.453172 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:19.454479 6469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:19.454595 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "25135c2052484e7d8704e3ebc7cc8f6c"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "483d2366001781f1edce13b013b6771d"
server_key_iv: "7c62fabceaf8b7851cda6e64d86d120d"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.454718 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:19.476847 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:19.480271 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:19.480453 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:19.487535 6469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43569
I20260927 00:39:19.487805 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43569 every 8 connection(s)
I20260927 00:39:19.488543 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:19.491387 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:19.496642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6469
I20260927 00:39:19.496759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:19.497005 392 external_mini_cluster.cc:1428] Setting key 6217094c2a3dabdbc7e4399a399c5d37
I20260927 00:39:19.497079 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Bootstrap starting.
I20260927 00:39:19.499588 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:19.500439 6537 log.cc:824] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:19.502735 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: No bootstrap required, opened a new log
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:19.507026 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } }
I20260927 00:39:19.507319 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:19.507380 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25135c2052484e7d8704e3ebc7cc8f6c, State: Initialized, Role: FOLLOWER
I20260927 00:39:19.507869 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } }
I20260927 00:39:19.508023 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:19.508088 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:19.508188 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:19.509274 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } }
I20260927 00:39:19.509634 6537 leader_election.cc:302] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25135c2052484e7d8704e3ebc7cc8f6c; no voters:
I20260927 00:39:19.509950 6537 leader_election.cc:288] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:19.510012 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:19.510239 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 LEADER]: Becoming Leader. State: Replica: 25135c2052484e7d8704e3ebc7cc8f6c, State: Running, Role: LEADER
I20260927 00:39:19.510533 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } }
I20260927 00:39:19.511329 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:19.511502 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:19.498425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.0.98.62:43569)
0927 00:39:19.498823 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:19.498839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:19.498882 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:39:19.499514 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:19.499532 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:19.499917 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:19.500237 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:19.501104 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.502209 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.503320 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.503806 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.503927 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:19.506617 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:19.506622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:19.506637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:19.506665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:19.508471 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:19.508906 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:19.508911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:19.508915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:19.509017 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:19.509273 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:19.509276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:19.509277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:19.509727 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:19.509881 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:19.510212 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 00:39:19.510528 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":102,"threads_started":1}
I20260927 00:39:19.512014 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } }
I20260927 00:39:19.512130 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:19.512383 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25135c2052484e7d8704e3ebc7cc8f6c. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } }
I20260927 00:39:19.512463 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:19.512729 6551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:19.516505 6551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:19.522951 6551 catalog_manager.cc:1347] Generated new cluster ID: 175a739dde8c46b692137ba1d9430c16
I20260927 00:39:19.523041 6551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:19.536149 6551 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:19.537174 6551 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:19.547756 6551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Generated new TSK 0
I20260927 00:39:19.548782 6551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:39:19.579831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:19.718914 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:19.719197 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:19.719244 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:19.723472 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:19.723562 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:19.723626 6563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:19.729745 6563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:19.730975 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:19.732376 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:19.742956 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.742949 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.743002 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:19.743072 6563 server_base.cc:1034] running on GCE node
I20260927 00:39:19.743723 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:19.744459 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:19.745610 6563 hybrid_clock.cc:648] HybridClock initialized: now 1790469559745584 us; error 41 us; skew 500 ppm
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:19.748554 6563 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:39:19.749521 6563 webserver.cc:460] Webserver started at http://127.0.98.1:45769/ using document root <none> and password file <none>
I20260927 00:39:19.750102 6563 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:19.750172 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:19.750396 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:19.752604 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "1b9fa94bd302b9354ab1fc6254780431"
server_key_iv: "f2844c7afd0c103f4b24ece0427763e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.753194 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "1b9fa94bd302b9354ab1fc6254780431"
server_key_iv: "f2844c7afd0c103f4b24ece0427763e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.757284 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:39:19.759955 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:19.761350 6563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:39:19.761468 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "1b9fa94bd302b9354ab1fc6254780431"
server_key_iv: "f2844c7afd0c103f4b24ece0427763e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.761571 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:19.779510 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:19.782830 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:19.783025 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:19.783625 6563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:19.784478 6563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:19.784528 6563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:19.784580 6563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:19.784603 6563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:19.792848 6563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34209
I20260927 00:39:19.792887 6691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34209 every 8 connection(s)
I20260927 00:39:19.793736 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:19.797698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6563
I20260927 00:39:19.797839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:19.798184 392 external_mini_cluster.cc:1428] Setting key 31b58361f928931f609bd6487e522e1b
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:19.809427 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:19.795615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39669 (local address 127.0.98.62:43569)
0927 00:39:19.795743 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:19.795746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:19.796720 (+ 974us) server_negotiation.cc:407] Connection header received
0927 00:39:19.798409 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:19.798412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:19.798464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:19.798540 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:19.799965 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.800675 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.802157 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.802480 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.802578 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:19.805691 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:19.805694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:19.805696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:19.805725 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:19.807117 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:19.807772 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:19.807775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:19.807776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:19.807856 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:19.808235 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:19.808240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:19.808242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:19.808448 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:19.808544 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:19.809120 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 00:39:19.809259 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:19.810420 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:19.795972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:39669)
0927 00:39:19.796537 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:19.796584 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:19.798052 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:19.798727 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:19.798734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:19.799196 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:19.799796 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:19.799817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.800814 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.800818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:19.802031 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:19.802042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:19.802663 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:19.802667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:19.802930 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:19.803583 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:19.803604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:19.805428 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:19.807246 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:19.807253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:19.807273 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:19.807656 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:19.807943 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:19.807946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:19.807947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:19.808093 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:19.808559 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:19.808567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:19.808879 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 00:39:19.809062 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:39:19.809470 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":209,"threads_started":1}
I20260927 00:39:19.811854 6692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:19.812100 6692 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:19.812682 6692 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
I20260927 00:39:19.814260 6501 ts_manager.cc:194] Registered new tserver with Master: 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:19.815420 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:39:19.827353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:19.942132 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:19.942425 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:19.942488 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:19.946887 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:19.946977 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:19.947065 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:19.953095 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:19.954252 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:19.955264 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:19.965335 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.965390 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:19.965354 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:19.965729 6697 server_base.cc:1034] running on GCE node
I20260927 00:39:19.966157 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:19.966811 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:19.967980 6697 hybrid_clock.cc:648] HybridClock initialized: now 1790469559967926 us; error 73 us; skew 500 ppm
Sep 27 00:39:19 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:19.970960 6697 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:39:19.971939 6697 webserver.cc:460] Webserver started at http://127.0.98.2:35205/ using document root <none> and password file <none>
I20260927 00:39:19.972575 6697 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:19.972647 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:19.972851 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:19.975023 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "f69de60d50bb446cbb09afa2638bf5bb"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "74f2ff4bd4e0a1d230e1e0572275b11e"
server_key_iv: "1b337edb414911917340995ad6189024"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.975632 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f69de60d50bb446cbb09afa2638bf5bb"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "74f2ff4bd4e0a1d230e1e0572275b11e"
server_key_iv: "1b337edb414911917340995ad6189024"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.979636 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 00:39:19.982352 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:19.983765 6697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:19.983873 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "f69de60d50bb446cbb09afa2638bf5bb"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "74f2ff4bd4e0a1d230e1e0572275b11e"
server_key_iv: "1b337edb414911917340995ad6189024"
server_key_version: "encryptionkey@0"
I20260927 00:39:19.983973 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.004030 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:20.007133 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:20.007292 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:20.007849 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:20.008739 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:20.008791 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.008832 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:20.008846 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.017745 6697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44685
I20260927 00:39:20.017800 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44685 every 8 connection(s)
I20260927 00:39:20.018658 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:20.024389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6697
I20260927 00:39:20.024488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:20.024709 392 external_mini_cluster.cc:1428] Setting key 5ed8d561feca8bf81acbca7d085f9b34
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:20.032902 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.020414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41667 (local address 127.0.98.62:43569)
0927 00:39:20.020542 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.020546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.021396 (+ 850us) server_negotiation.cc:407] Connection header received
0927 00:39:20.022766 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.022768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.022825 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.022897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.024098 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.024732 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.026060 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.026356 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.026465 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.029132 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.029136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.029138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.029167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.030761 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.031273 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.031277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.031279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.031379 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.031678 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.031680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.031682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.031927 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.032019 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.032589 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.032729 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:20.033828 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.020826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:41667)
0927 00:39:20.021249 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.021294 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.022556 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.023102 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.023109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.023463 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.023958 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.023972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.024924 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.024926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.025949 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.025960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.026514 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.026517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.026663 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.027238 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.027255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.028951 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.030889 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.030894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.030909 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.031177 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.031465 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.031467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.031468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.031598 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.032022 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.032027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.032322 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.032517 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.032805 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260927 00:39:20.035087 6826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:20.035357 6826 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:20.035933 6826 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:20.037250 6501 ts_manager.cc:194] Registered new tserver with Master: f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685)
I20260927 00:39:20.037909 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:39:20.056048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:20.169708 6831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:20.170121 6831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:20.170186 6831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:20.174435 6831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:20.174520 6831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:20.174606 6831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:20.180085 6831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:20.181252 6831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:20.182312 6831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:20.192301 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.192343 6831 server_base.cc:1034] running on GCE node
W20260927 00:39:20.192318 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:20.192332 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.192962 6831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:20.193603 6831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:20.194782 6831 hybrid_clock.cc:648] HybridClock initialized: now 1790469560194762 us; error 34 us; skew 500 ppm
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:20.197674 6831 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:39:20.198549 6831 webserver.cc:460] Webserver started at http://127.0.98.3:32863/ using document root <none> and password file <none>
I20260927 00:39:20.199110 6831 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.199172 6831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.199357 6831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:20.201488 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "55031aae518234f7102b98fe77190ecc"
server_key_iv: "31b8b9049876fb15a63b7fa9b9664766"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.202057 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "55031aae518234f7102b98fe77190ecc"
server_key_iv: "31b8b9049876fb15a63b7fa9b9664766"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.206049 6831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:39:20.208716 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.210108 6831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:20.210233 6831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "55031aae518234f7102b98fe77190ecc"
server_key_iv: "31b8b9049876fb15a63b7fa9b9664766"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.210341 6831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.239372 6831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:20.242939 6831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:20.243116 6831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:20.243697 6831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:20.244603 6831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:20.244655 6831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.244697 6831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:20.244729 6831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.253851 6831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37465
I20260927 00:39:20.253988 6959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37465 every 8 connection(s)
I20260927 00:39:20.254820 6831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:20.262940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6831
I20260927 00:39:20.263068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:20.263329 392 external_mini_cluster.cc:1428] Setting key 7f2930847ba81edd3a01b2d45d3324e6
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:20.268909 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.256815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52637 (local address 127.0.98.62:43569)
0927 00:39:20.257062 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.257066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.258030 (+ 964us) server_negotiation.cc:407] Connection header received
0927 00:39:20.259442 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.259445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.259503 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.259576 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.260853 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.261388 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.262533 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.262826 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.262942 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.265554 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.265559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.265562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.265593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.266863 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.267359 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.267362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.267364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.267440 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.267756 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.267759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.267761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.267954 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.268037 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.268616 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.268745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 00:39:20.269868 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.257250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:52637)
0927 00:39:20.257766 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.257912 (+ 146us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.259205 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.259755 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.259762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.260150 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.260707 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.260720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.261511 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.261513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.262407 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.262417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.263028 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.263033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.263222 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.263891 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.263908 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.265365 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.266976 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.266983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.267001 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.267262 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.267523 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.267526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.267528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.267671 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.268048 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.268055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.268385 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.268566 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.268939 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260927 00:39:20.271080 6960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:20.271366 6960 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:20.271987 6960 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
I20260927 00:39:20.273187 6501 ts_manager.cc:194] Registered new tserver with Master: 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
I20260927 00:39:20.273871 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52637
I20260927 00:39:20.277565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 00:39:20.303910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36723
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43569,127.0.98.61:36723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:20.415827 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:20.416111 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:20.416154 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:20.420265 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:20.420352 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:20.420372 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:20.420389 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:20.420413 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:20.425822 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43569,127.0.98.61:36723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
W20260927 00:39:20.426049 6965 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43569,127.0.98.61:36723'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:39:20.427024 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:20.428073 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:20.437233 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:20.437368 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.437480 6965 server_base.cc:1034] running on GCE node
W20260927 00:39:20.437496 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.438202 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:20.439402 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:20.440604 6965 hybrid_clock.cc:648] HybridClock initialized: now 1790469560440538 us; error 89 us; skew 500 ppm
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:20.443933 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:39:20.444944 6965 webserver.cc:460] Webserver started at http://127.0.98.61:40869/ using document root <none> and password file <none>
I20260927 00:39:20.445503 6965 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.445564 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.445778 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:20.447962 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/instance:
uuid: "b74d423ad6374397bca360fb094c16bd"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "eea6e1cf6b43476db826fa09eda6e298"
server_key_iv: "d0bb0e03b7faa0c8e162855d8b671fcd"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.448537 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal/instance:
uuid: "b74d423ad6374397bca360fb094c16bd"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "eea6e1cf6b43476db826fa09eda6e298"
server_key_iv: "d0bb0e03b7faa0c8e162855d8b671fcd"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.452520 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:20.455132 6980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.456557 6965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:20.456672 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "b74d423ad6374397bca360fb094c16bd"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "eea6e1cf6b43476db826fa09eda6e298"
server_key_iv: "d0bb0e03b7faa0c8e162855d8b671fcd"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.456768 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.480304 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:20.483343 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:20.483528 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:20.491099 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36723
I20260927 00:39:20.491345 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36723 every 8 connection(s)
I20260927 00:39:20.492318 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:39:20.495188 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:20.498967 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:39:20.500731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 6965
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:20.512710 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.500850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.98.62:43569)
0927 00:39:20.500995 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.501000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.501967 (+ 967us) server_negotiation.cc:407] Connection header received
0927 00:39:20.502915 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.502918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.502965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.503041 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.504084 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.504902 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.506019 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.506234 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.506327 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.508791 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.508796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.508799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.508828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.510494 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.511122 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.511125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.511126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.511201 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.511511 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.511514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.511516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.511771 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.511857 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.512416 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.512545 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:39:20.513643 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.501474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.0.98.61:36723)
0927 00:39:20.501838 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.501855 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.501968 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:39:20.502695 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.502712 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.502920 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.503090 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.503976 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.505180 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.505995 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.506362 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.506497 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.509196 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.509203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.509218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.509241 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.511000 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.511407 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.511411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.511412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.511488 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.511755 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.511758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.511760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.512178 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.512440 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.512615 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.512803 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260927 00:39:20.513640 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.501307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60376)
0927 00:39:20.501831 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.501882 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.502707 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.503079 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.503083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.503701 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.503955 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.503969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.505041 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.505044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.505873 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.505882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.506373 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.506376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.506518 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.507171 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.507187 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.508647 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.510608 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.510614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.510623 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.511012 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.511277 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.511281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.511283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.511423 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.511877 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.511884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.512182 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.512363 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.512727 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":89,"spinlock_wait_cycles":768,"thread_start_us":154,"threads_started":1}
I20260927 00:39:20.514878 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:20.522641 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.515344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37006)
0927 00:39:20.515475 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.515489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.515598 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.515868 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.515870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.516010 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.516089 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.516093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.516847 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.516850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.517483 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.517493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.517927 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.517930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.518003 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.518390 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.518399 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.519614 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.521325 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.521327 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.521329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.521525 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.521787 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.521789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.521790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.521867 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.522185 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.522186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.522262 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.522333 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.522474 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 00:39:20.522651 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.515411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.98.61:36723)
0927 00:39:20.515494 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.515498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.515550 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:39:20.515673 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.515675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.515709 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.515777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.516186 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.516727 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.517595 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.517814 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.517897 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.519792 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.519795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.519797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.519817 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.521136 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.521612 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.521615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.521616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.521683 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.521946 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.521948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.521949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.522121 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.522191 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.522392 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.522527 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":39}
I20260927 00:39:20.525924 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Bootstrap starting.
I20260927 00:39:20.528581 7033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:20.529340 7033 log.cc:824] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:20.531339 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: No bootstrap required, opened a new log
I20260927 00:39:20.534430 7033 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } }
I20260927 00:39:20.534664 7033 raft_consensus.cc:374] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:20.534729 7033 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74d423ad6374397bca360fb094c16bd, State: Initialized, Role: FOLLOWER
I20260927 00:39:20.535179 7033 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } }
I20260927 00:39:20.535699 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } } }
I20260927 00:39:20.535888 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:20.536422 7033 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:39:20.541287 7050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:39:20.541388 7050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:39:20.543438 6965 master_runner.cc:428] Detected that this master b74d423ad6374397bca360fb094c16bd is joining an existing cluster
I20260927 00:39:20.543514 6965 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36723
I20260927 00:39:20.555171 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.546480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37010)
0927 00:39:20.546751 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.546762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.546867 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.547164 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.547167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.547471 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.547693 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.547702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.548757 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.548759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.549377 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.549388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.549892 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.549895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.549978 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.550968 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.550979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.551439 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.553282 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.553308 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.553311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.553561 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.553850 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.553853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.553855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.553951 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.554597 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.554600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.554698 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.554799 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.554958 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260927 00:39:20.555696 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.546559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.0.98.61:36723)
0927 00:39:20.546668 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.546672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.546817 (+ 145us) server_negotiation.cc:407] Connection header received
0927 00:39:20.546948 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.546951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.546988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.547027 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.547816 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.548655 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.549482 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.549679 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.549767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.551546 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.551549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.551552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.551573 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.553182 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.553667 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.553671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.553673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.553758 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.554041 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.554044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.554046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.554271 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.554367 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.554950 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.555071 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:39:20.555950 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.546144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.0.98.62:43569)
0927 00:39:20.546283 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.546288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.546814 (+ 526us) server_negotiation.cc:407] Connection header received
0927 00:39:20.546958 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.546961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.547014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.547096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.548807 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.549613 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.550615 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.550973 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.551086 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.552379 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.552383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.552386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.552419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.553968 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.554377 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.554381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.554383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.554478 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.554940 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.554944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.554946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.555188 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.555267 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.555659 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.555789 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:39:20.555917 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.546041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60384)
0927 00:39:20.546719 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.546733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.546848 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.547153 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.547155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.547493 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.547693 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.547702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.549711 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.549714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.550452 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.550463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.551094 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.551097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.551189 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.551780 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.551789 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.552157 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.554069 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.554072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.554074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.554275 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.554628 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.554631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.554632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.554725 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.555367 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.555370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.555473 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.555572 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.555719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":48,"spinlock_wait_cycles":11264,"thread_start_us":121,"threads_started":1}
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:20.571879 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.561306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.0.98.61:36723)
0927 00:39:20.561447 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.561451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.561781 (+ 330us) server_negotiation.cc:407] Connection header received
0927 00:39:20.562054 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.562057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.562106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.562173 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.562873 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.563379 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.564657 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.564929 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.565124 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.568115 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.568120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.568123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.568152 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.569843 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.570450 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.570454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.570456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.570551 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.570852 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.570855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.570857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.571081 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.571175 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.571569 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.571699 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:39:20.571945 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.561293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37014)
0927 00:39:20.561660 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.561705 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.561853 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.562225 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.562232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.562623 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.562752 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.562759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.563490 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.563494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.564512 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.564527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.565135 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.565139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.565279 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.566136 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.566152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.567958 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.570002 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.570017 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.570025 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.570349 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.570644 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.570647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.570649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.570748 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.571185 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.571193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.571352 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.571510 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.571752 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
I20260927 00:39:20.574132 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36723: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b74d423ad6374397bca360fb094c16bd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36723
}
attrs {
promote: true
}
}
dest_uuid: "25135c2052484e7d8704e3ebc7cc8f6c"
cas_config_opid_index: -1
I20260927 00:39:20.574875 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:39:20.578218 7007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:20.578686 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b74d423ad6374397bca360fb094c16bd (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 00:39:20.580566 7007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:39:20.582024 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36723
I20260927 00:39:20.582185 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:20.582417 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } }
I20260927 00:39:20.582535 6965 master.cc:537] Master@127.0.98.61:36723 shutting down...
I20260927 00:39:20.582623 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
W20260927 00:39:20.586752 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:39:20.587728 6965 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:20.588151 6965 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:20.588312 6965 tablet_replica.cc:330] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 00:39:20.600134 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.599813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37026)
0927 00:39:20.599956 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.600037 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:39:20.600389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42817
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:20.600437 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c -> Peer b74d423ad6374397bca360fb094c16bd (127.0.98.61:36723): Couldn't send request to peer b74d423ad6374397bca360fb094c16bd. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:39:20.604143 6965 master.cc:552] Master@127.0.98.61:36723 shutdown complete.
I20260927 00:39:20.604236 6965 master_runner.cc:305] Clearing existing system tablet
I20260927 00:39:20.604490 6965 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.604544 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.605883 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:20.606834 7069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.607069 6965 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.607195 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:20.607239 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "b74d423ad6374397bca360fb094c16bd"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "eea6e1cf6b43476db826fa09eda6e298"
server_key_iv: "d0bb0e03b7faa0c8e162855d8b671fcd"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.609558 6965 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:20.612180 6965 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:39:20.612303 6965 log.cc:1196] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:39:20.612571 6965 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Deleting consensus metadata
I20260927 00:39:20.612782 6965 master_runner.cc:315] Copying system tablet from 127.0.98.62:43569
I20260927 00:39:20.613988 6965 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43569
I20260927 00:39:20.624853 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.614666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60398)
0927 00:39:20.616447 (+ 1781us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.616464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.616656 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.616985 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.616988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.617215 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.617350 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.617358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.618435 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.618439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.619365 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.619377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.620069 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.620072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.620175 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.620786 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.620801 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.621223 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.623208 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.623211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.623214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.623478 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.623759 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.623762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.623764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.623854 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.624269 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.624272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.624379 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.624490 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.624638 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"thread_start_us":101,"threads_started":1}
I20260927 00:39:20.624851 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.614755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.98.62:43569)
0927 00:39:20.614884 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.614889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.616564 (+ 1675us) server_negotiation.cc:407] Connection header received
0927 00:39:20.616791 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.616795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.616857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.616955 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.617461 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.618303 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.619558 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.619946 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.620054 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.621393 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.621397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.621400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.621430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.623088 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.623577 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.623581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.623583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.623672 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.623933 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.623936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.623938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.624154 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.624270 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.624566 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.624690 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:39:20.625845 6521 tablet_copy_service.cc:140] P 25135c2052484e7d8704e3ebc7cc8f6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b74d423ad6374397bca360fb094c16bd ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60398)
I20260927 00:39:20.626026 6521 tablet_copy_service.cc:161] P 25135c2052484e7d8704e3ebc7cc8f6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b74d423ad6374397bca360fb094c16bd at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60398: session id = b74d423ad6374397bca360fb094c16bd-00000000000000000000000000000000
I20260927 00:39:20.627662 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:39:20.629415 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:39:20.631732 6965 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: tablet copy: Starting download of 0 data blocks...
I20260927 00:39:20.631922 6965 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: tablet copy: Starting download of 1 WAL segments...
I20260927 00:39:20.633942 6965 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:39:20.635866 6521 tablet_copy_service.cc:342] P 25135c2052484e7d8704e3ebc7cc8f6c: Request end of tablet copy session b74d423ad6374397bca360fb094c16bd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60398
I20260927 00:39:20.635999 6521 tablet_copy_service.cc:433] P 25135c2052484e7d8704e3ebc7cc8f6c: ending tablet copy session b74d423ad6374397bca360fb094c16bd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b74d423ad6374397bca360fb094c16bd
I20260927 00:39:20.643797 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:20.645730 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:20.646088 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.646253 6965 server_base.cc:1034] running on GCE node
W20260927 00:39:20.646302 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.646613 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:20.646914 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:20.648053 6965 hybrid_clock.cc:648] HybridClock initialized: now 1790469560648043 us; error 31 us; skew 500 ppm
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:20.649761 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:39:20.650244 6965 webserver.cc:460] Webserver started at http://127.0.98.61:39189/ using document root <none> and password file <none>
I20260927 00:39:20.650429 6965 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.650470 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.651871 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:20.652979 7086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.653249 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:20.653326 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "b74d423ad6374397bca360fb094c16bd"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "eea6e1cf6b43476db826fa09eda6e298"
server_key_iv: "d0bb0e03b7faa0c8e162855d8b671fcd"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.653409 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.672187 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:20.674383 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:20.678941 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36723
I20260927 00:39:20.678980 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36723 every 8 connection(s)
I20260927 00:39:20.679517 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:39:20.681314 7139 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:39:20.682348 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Bootstrap starting.
I20260927 00:39:20.692123 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:39:20.692453 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: Bootstrap complete.
I20260927 00:39:20.693044 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } }
I20260927 00:39:20.693193 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b74d423ad6374397bca360fb094c16bd, State: Initialized, Role: LEARNER
I20260927 00:39:20.693305 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } }
I20260927 00:39:20.693768 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:20.693660 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } }
I20260927 00:39:20.694006 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: LEARNER
I20260927 00:39:20.696967 7153 catalog_manager.cc:1259] Loaded cluster ID: 175a739dde8c46b692137ba1d9430c16
I20260927 00:39:20.697077 7153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: loading cluster ID for follower catalog manager: success
I20260927 00:39:20.699388 7153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: acquiring CA information for follower catalog manager: success
I20260927 00:39:20.701061 7153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:39:20.733862 7067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:20.734153 7067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:20.734194 7067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:20.738442 7067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:20.738513 7067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:20.738533 7067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:20.738548 7067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:20.738564 7067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:20.744079 7067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:20.745194 7067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:20.746220 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:20.755533 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:20.755681 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:20.755614 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:20.755554 7067 server_base.cc:1034] running on GCE node
I20260927 00:39:20.756726 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:20.758005 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:20.759208 7067 hybrid_clock.cc:648] HybridClock initialized: now 1790469560759186 us; error 38 us; skew 500 ppm
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:20.762322 7067 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:39:20.763327 7067 webserver.cc:460] Webserver started at http://127.0.98.60:45327/ using document root <none> and password file <none>
I20260927 00:39:20.763917 7067 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.763990 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.764182 7067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:20.766554 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/instance:
uuid: "d62f7363a6d84930a5bfef65fc465bc4"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "bb2ca7c77ddcca09b37d94506e7e6551"
server_key_iv: "b937af151b943dc134f051bbd503d3bb"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.767134 7067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal/instance:
uuid: "d62f7363a6d84930a5bfef65fc465bc4"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "bb2ca7c77ddcca09b37d94506e7e6551"
server_key_iv: "b937af151b943dc134f051bbd503d3bb"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.771203 7067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:20.774121 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.775614 7067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:20.775771 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "d62f7363a6d84930a5bfef65fc465bc4"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "bb2ca7c77ddcca09b37d94506e7e6551"
server_key_iv: "b937af151b943dc134f051bbd503d3bb"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.775894 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.804798 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:20.808135 7067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:20.808344 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:20.816015 7067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42817
I20260927 00:39:20.816318 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42817 every 8 connection(s)
I20260927 00:39:20.817085 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:39:20.817844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7067
I20260927 00:39:20.817916 6692 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:20.820495 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:20.825472 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:20.834227 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.819603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.98.60:42817)
0927 00:39:20.820035 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.820053 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.820098 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:39:20.820980 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.821005 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.821493 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.821877 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.822619 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.824250 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.825142 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.825862 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.826022 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.828899 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.828906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.828919 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.828950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.830862 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.831342 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.831350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.831357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.831507 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.831800 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.831802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.831804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.832287 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.832463 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.832793 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.833171 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":1792,"thread_start_us":116,"threads_started":1}
I20260927 00:39:20.837107 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.827573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.98.62:43569)
0927 00:39:20.827708 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.827713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.828053 (+ 340us) server_negotiation.cc:407] Connection header received
0927 00:39:20.828197 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.828224 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.828290 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.828369 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.829176 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.829740 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.830886 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.831110 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.831198 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.833711 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.833715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.833717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.833743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.835189 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.835661 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.835665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.835666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.835768 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.836133 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.836136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.836138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.836404 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.836504 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.836875 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.836985 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:39:20.837157 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.827415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60406)
0927 00:39:20.827909 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.827964 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.828100 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.828440 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.828446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.828855 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.829022 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.829032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.829977 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.829979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.830759 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.830770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.831215 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.831219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.831333 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.831940 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.831951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.833543 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.835297 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.835305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.835311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.835554 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.835925 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.835928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.835929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.836039 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.836534 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.836541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.836673 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.836820 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.836987 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"spinlock_wait_cycles":4992,"thread_start_us":178,"threads_started":1}
I20260927 00:39:20.838310 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:20.846892 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.838888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37036)
0927 00:39:20.839018 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.839032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.839115 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.839478 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.839480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.839620 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.839726 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.839731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.840712 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.840715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.841281 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.841290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.841882 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.841885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.841982 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.842432 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.842444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.843647 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.845595 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.845598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.845600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.845775 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.846038 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.846040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.846041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.846115 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.846486 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.846488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.846569 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.846647 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.846758 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:39:20.846925 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.839007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.98.61:36723)
0927 00:39:20.839282 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.839285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.839296 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:39:20.839332 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.839334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.839373 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.839449 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.839838 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.840603 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.841431 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.841679 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.841878 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.843852 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.843856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.843858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.843885 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.845493 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.845865 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.845868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.845869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.845956 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.846213 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.846216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.846217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.846404 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.846489 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.846689 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.846790 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260927 00:39:20.847724 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469560, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:39:20.855504 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.848278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.98.60:42817)
0927 00:39:20.848361 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.848364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.848391 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:39:20.848542 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.848545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.848578 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.848636 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.848994 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.849834 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.850536 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.850745 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.850827 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.852709 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.852713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.852716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.852738 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.853984 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.854436 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.854439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.854441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.854507 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.854723 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.854726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.854727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.854908 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.854988 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.855208 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.855355 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":31}
I20260927 00:39:20.855504 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.848195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.0.1:35020)
0927 00:39:20.848336 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.848350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.848469 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.848652 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.848655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.848798 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.848861 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.848866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.849930 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.849933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.850437 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.850444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.850834 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.850836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.850946 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.851350 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.851360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.852587 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.854078 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.854081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.854083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.854345 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.854580 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.854583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.854585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.854655 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.854986 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.854989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.855076 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.855158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.855308 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":30}
I20260927 00:39:20.858475 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Bootstrap starting.
I20260927 00:39:20.860857 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:20.861598 7221 log.cc:824] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:20.863535 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: No bootstrap required, opened a new log
I20260927 00:39:20.867177 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } }
I20260927 00:39:20.867411 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:20.867463 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62f7363a6d84930a5bfef65fc465bc4, State: Initialized, Role: FOLLOWER
I20260927 00:39:20.867904 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } }
I20260927 00:39:20.868383 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } } }
I20260927 00:39:20.868534 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:20.869130 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:39:20.873816 7238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:39:20.873920 7238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:39:20.877838 7067 master_runner.cc:428] Detected that this master d62f7363a6d84930a5bfef65fc465bc4 is joining an existing cluster
I20260927 00:39:20.877933 7067 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42817
I20260927 00:39:20.887883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6563
I20260927 00:39:20.893939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34209
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43569
--webserver_port=45769
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:39:20.916297 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.880810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.98.60:42817)
0927 00:39:20.880917 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.880920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.881274 (+ 354us) server_negotiation.cc:407] Connection header received
0927 00:39:20.881448 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.881451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.881491 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.881537 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.882183 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.882879 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.883943 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.884341 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.884446 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.885704 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.885708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.885711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.885733 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.914567 (+ 28834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915075 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.915078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.915080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.915146 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915379 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.915382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.915384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.915610 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.915673 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.915958 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.916077 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":896}
I20260927 00:39:20.916625 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.881664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37052)
0927 00:39:20.881922 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.881934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.882064 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.882399 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.882402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.882592 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.882688 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.882694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.883741 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.883744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.884525 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.884534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.885466 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.885482 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.885597 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.886388 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.886400 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.912882 (+ 26482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.914964 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.914967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.914970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.915218 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.915564 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.915569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.915653 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.916069 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.916073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.916160 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.916301 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.916449 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260927 00:39:20.916759 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.880640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.0.1:35036)
0927 00:39:20.881197 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.881212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.881342 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.881669 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.881672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.881940 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.882088 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.882095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.882987 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.882989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.883846 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.883853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.884472 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.884476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.884563 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.885046 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.885060 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.885561 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.914732 (+ 29171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.914735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.914738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.914981 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.915231 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.915236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.915304 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.915724 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.915727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.915799 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.916441 (+ 642us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.916568 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":117,"threads_started":1}
I20260927 00:39:20.917196 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.881440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.98.61:36723)
0927 00:39:20.881568 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.881572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.882005 (+ 433us) server_negotiation.cc:407] Connection header received
0927 00:39:20.882164 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.882167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.882216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.882265 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.882814 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.883606 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.884650 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.885238 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.885319 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.913165 (+ 27846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.913170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.913173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.913225 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.914794 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915366 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.915369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.915371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.915454 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.915742 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.915745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.915747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.915946 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.916029 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.916885 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.917038 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:39:20.920925 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.880650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.0.98.62:43569)
0927 00:39:20.880851 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.880856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.881399 (+ 543us) server_negotiation.cc:407] Connection header received
0927 00:39:20.881595 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.881598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.881651 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.881705 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.882367 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.883173 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.884758 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.885119 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.885194 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.917953 (+ 32759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.917958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.917961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.917992 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.919385 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.919771 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.919775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.919776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.919852 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.920098 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.920101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.920103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.920337 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.920427 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.920670 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.920790 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 00:39:20.920931 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.880993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60422)
0927 00:39:20.881310 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.881324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.881469 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.881849 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.881851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.882091 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.882219 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.882226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.883311 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.883314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.884636 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.884644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.885342 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.885346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.885435 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.885891 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.885909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.917763 (+ 31854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.919494 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.919497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.919499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.919679 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.919932 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.919935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.919937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.920019 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.920423 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.920426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.920518 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.920602 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.920747 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":27,"thread_start_us":125,"threads_started":1}
Sep 27 00:39:20 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469559, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:39:20.932070 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.924642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.0.1:35050)
0927 00:39:20.924747 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.924762 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.924839 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.925229 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.925232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.925417 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.925537 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.925544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.926311 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.926315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.926866 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.926874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.927293 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.927297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.927389 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.927868 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.927882 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.929107 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.930667 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.930671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.930674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.930861 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.931104 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.931107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.931109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.931188 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.931540 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.931543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.931638 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.931723 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.931878 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 00:39:20.932070 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.924807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.98.60:42817)
0927 00:39:20.924913 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.924917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.924927 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:39:20.924993 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.924996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.925044 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.925112 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.925666 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.926205 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.926970 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.927199 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.927284 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.929240 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.929244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.929247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.929272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.930560 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.930947 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.930950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.930952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.931024 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.931262 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.931266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.931267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.931446 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.931539 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.931782 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.931885 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:39:20.933348 6501 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42817: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42817
}
attrs {
promote: true
}
}
dest_uuid: "25135c2052484e7d8704e3ebc7cc8f6c"
cas_config_opid_index: 5
I20260927 00:39:20.933880 6501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } }
I20260927 00:39:20.934875 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.935998 7195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:20.936383 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42817
I20260927 00:39:20.936559 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.936647 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:20.937351 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:39:20.938052 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:20.938691 7067 master.cc:537] Master@127.0.98.60:42817 shutting down...
W20260927 00:39:20.940121 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c -> Peer d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60:42817): Couldn't send request to peer d62f7363a6d84930a5bfef65fc465bc4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:39:20.947484 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.935081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37056)
0927 00:39:20.941623 (+ 6542us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.941640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.941740 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.943283 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.943287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.943311 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:20.943423 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.943429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.944523 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.944527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.945816 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.945830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.947002 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.947006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.947105 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.947166 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.947236 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.947326 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6087}
I20260927 00:39:20.947952 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.935321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.0.98.61:36723)
0927 00:39:20.940689 (+ 5368us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.940693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.941824 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 00:39:20.941884 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.941887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.941950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.942114 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:20.943572 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.944381 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.945974 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.946640 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.946748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.946926 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.947753 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.947842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5266}
I20260927 00:39:20.948457 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 LEARNER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:39:20.948900 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:39:20.950856 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.952840 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25135c2052484e7d8704e3ebc7cc8f6c. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.952962 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: LEARNER
I20260927 00:39:20.953063 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: true } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.953119 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: LEARNER
I20260927 00:39:20.954663 7061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: attempting to promote NON_VOTER b74d423ad6374397bca360fb094c16bd to VOTER
I20260927 00:39:20.955142 7061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } }
I20260927 00:39:20.956333 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 LEARNER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:39:20.956745 7064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 00:39:20.957196 7067 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:20.957504 7067 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:20.957617 7067 tablet_replica.cc:330] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: stopping tablet replica
W20260927 00:39:20.959172 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c -> Peer d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60:42817): Couldn't send request to peer d62f7363a6d84930a5bfef65fc465bc4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:39:20.960302 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b74d423ad6374397bca360fb094c16bd (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.961776 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.961882 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:20.962003 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.962061 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:20.962811 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b74d423ad6374397bca360fb094c16bd (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.965209 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:20.965369 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:20.976651 7067 master.cc:552] Master@127.0.98.60:42817 shutdown complete.
I20260927 00:39:20.976713 7067 master_runner.cc:305] Clearing existing system tablet
I20260927 00:39:20.976931 7067 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:20.976981 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:20.978127 7067 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 00:39:20.978880 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:20.979243 7067 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:20.979363 7067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:20.979401 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "d62f7363a6d84930a5bfef65fc465bc4"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "bb2ca7c77ddcca09b37d94506e7e6551"
server_key_iv: "b937af151b943dc134f051bbd503d3bb"
server_key_version: "encryptionkey@0"
I20260927 00:39:20.981104 7067 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:20.982995 7067 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:39:20.983085 7067 log.cc:1196] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:39:20.983304 7067 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Deleting consensus metadata
I20260927 00:39:20.983567 7067 master_runner.cc:315] Copying system tablet from 127.0.98.62:43569
I20260927 00:39:20.984717 7067 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43569
I20260927 00:39:20.994820 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.987439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.0.98.62:43569)
0927 00:39:20.987574 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:20.987578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:20.987833 (+ 255us) server_negotiation.cc:407] Connection header received
0927 00:39:20.987963 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:20.987967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:20.988015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:20.988111 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:20.988773 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.989418 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.990242 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.990467 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.990566 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.991662 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:20.991666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:20.991668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:20.991695 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:20.993080 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.993527 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.993531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.993533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.993618 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:20.993860 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:20.993863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:20.993865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:20.994153 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:20.994283 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:20.994536 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 00:39:20.994651 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:39:20.995131 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:20.987357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60434)
0927 00:39:20.987724 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:20.987738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:20.987867 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:20.988172 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:20.988175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:20.988537 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:20.988647 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.988654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.989527 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.989531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.990128 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:20.990137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:20.990582 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:20.990586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:20.990677 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:20.991159 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:20.991175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:20.991513 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:20.993209 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.993212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.993213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.993433 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.993699 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:20.993702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:20.993704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:20.993784 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:20.994271 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:20.994275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:20.994379 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 00:39:20.994470 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:39:20.994683 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
I20260927 00:39:20.995541 6521 tablet_copy_service.cc:140] P 25135c2052484e7d8704e3ebc7cc8f6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d62f7363a6d84930a5bfef65fc465bc4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60434)
I20260927 00:39:20.995648 6521 tablet_copy_service.cc:161] P 25135c2052484e7d8704e3ebc7cc8f6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d62f7363a6d84930a5bfef65fc465bc4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60434: session id = d62f7363a6d84930a5bfef65fc465bc4-00000000000000000000000000000000
I20260927 00:39:20.996558 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:39:20.997736 7067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:39:20.999619 7067 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: tablet copy: Starting download of 0 data blocks...
I20260927 00:39:20.999817 7067 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: tablet copy: Starting download of 1 WAL segments...
I20260927 00:39:21.001278 7067 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:39:21.002907 6521 tablet_copy_service.cc:342] P 25135c2052484e7d8704e3ebc7cc8f6c: Request end of tablet copy session d62f7363a6d84930a5bfef65fc465bc4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60434
I20260927 00:39:21.002993 6521 tablet_copy_service.cc:433] P 25135c2052484e7d8704e3ebc7cc8f6c: ending tablet copy session d62f7363a6d84930a5bfef65fc465bc4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d62f7363a6d84930a5bfef65fc465bc4
I20260927 00:39:21.010819 7067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:21.012774 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.013262 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.016772 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:21.017534 7067 server_base.cc:1034] running on GCE node
I20260927 00:39:21.017840 7067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:21.018181 7067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:21.019337 7067 hybrid_clock.cc:648] HybridClock initialized: now 1790469561019321 us; error 36 us; skew 500 ppm
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:21.020821 7067 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:39:21.021255 7067 webserver.cc:460] Webserver started at http://127.0.98.60:38523/ using document root <none> and password file <none>
I20260927 00:39:21.021438 7067 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:21.021472 7067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:21.022559 7067 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:21.023376 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.023591 7067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:21.023658 7067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "d62f7363a6d84930a5bfef65fc465bc4"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "bb2ca7c77ddcca09b37d94506e7e6551"
server_key_iv: "b937af151b943dc134f051bbd503d3bb"
server_key_version: "encryptionkey@0"
I20260927 00:39:21.023736 7067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:21.028461 7067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:21.030347 7067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:21.035076 7067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42817
I20260927 00:39:21.035079 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42817 every 8 connection(s)
I20260927 00:39:21.035641 7067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:39:21.037144 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:39:21.038013 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Bootstrap starting.
I20260927 00:39:21.040262 6826 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:21.048425 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:39:21.048738 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Bootstrap complete.
I20260927 00:39:21.049343 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } }
I20260927 00:39:21.049465 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d62f7363a6d84930a5bfef65fc465bc4, State: Initialized, Role: LEARNER
I20260927 00:39:21.049568 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } }
I20260927 00:39:21.049902 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:21.050046 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:21.050107 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:39:21.052873 7338 catalog_manager.cc:1259] Loaded cluster ID: 175a739dde8c46b692137ba1d9430c16
I20260927 00:39:21.053079 7338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: loading cluster ID for follower catalog manager: success
I20260927 00:39:21.055089 7338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: acquiring CA information for follower catalog manager: success
W20260927 00:39:21.055418 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:21.055747 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:21.055809 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:39:21.056460 7338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:39:21.062095 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:21.062184 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:21.062273 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:21.069226 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45769
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:21.070307 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:21.071309 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:21.081038 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.081044 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.081044 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:21.081619 7247 server_base.cc:1034] running on GCE node
I20260927 00:39:21.081982 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:21.082630 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:21.083788 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790469561083750 us; error 54 us; skew 500 ppm
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:21.086575 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:39:21.087627 7247 webserver.cc:460] Webserver started at http://127.0.98.1:45769/ using document root <none> and password file <none>
I20260927 00:39:21.088172 7247 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:21.088286 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:21.092605 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:39:21.094981 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.096498 7247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:21.096621 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "1b9fa94bd302b9354ab1fc6254780431"
server_key_iv: "f2844c7afd0c103f4b24ece0427763e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:21.097185 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:21.110888 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:21.114086 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:21.114243 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:21.114791 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:21.115629 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:21.115674 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.115711 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:21.115736 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.124919 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34209
I20260927 00:39:21.124950 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34209 every 8 connection(s)
I20260927 00:39:21.126004 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:21.136328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7247
I20260927 00:39:21.136535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6697
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:21.144487 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.130347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55031 (local address 127.0.98.61:36723)
0927 00:39:21.130470 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.130473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.130525 (+ 52us) server_negotiation.cc:407] Connection header received
0927 00:39:21.131046 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.131049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.131098 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.131177 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.134023 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.134514 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135443 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.135774 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135882 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.140001 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.140005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.140007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.140034 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.141596 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142304 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.142308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.142310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.142392 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142989 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.142992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.142994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.143245 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.143339 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.144155 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.144323 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:39:21.144486 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.128310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59455 (local address 127.0.98.62:43569)
0927 00:39:21.128609 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.128614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.129724 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 00:39:21.130948 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.130951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.131016 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.131069 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.132795 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.133681 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135476 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.135821 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135917 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.139992 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.139996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.139999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.140030 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.141904 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142691 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.142694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.142696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.142805 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.143309 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.143313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.143315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.143542 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.143623 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.144157 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.144323 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260927 00:39:21.144752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44685
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43569
--webserver_port=35205
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:39:21.145704 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.130192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.98.1:55031)
0927 00:39:21.130453 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.130465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.130710 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.131396 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.131405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.131931 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.132611 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.132628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.134622 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.134626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.135319 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.135328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.136014 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.136017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.136251 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.137085 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.137119 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.139809 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.141747 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.141754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.141775 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.142202 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.142669 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.142674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.142869 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.143396 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.143403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.143857 (+ 454us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.145308 (+ 1451us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.145458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":255,"spinlock_wait_cycles":33024,"thread_start_us":92,"threads_started":1}
I20260927 00:39:21.145959 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.128922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.98.1:46129)
0927 00:39:21.129870 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.129884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.130701 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.131396 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.131406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.131955 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.132619 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.132629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.133995 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.133999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.135334 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.135341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.135871 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.136073 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.137085 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.137108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.139125 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.142066 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.142071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.142341 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.142905 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.142909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.142993 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.143435 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.143438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.143857 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.144077 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.144523 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":290,"spinlock_wait_cycles":106752,"thread_start_us":181,"threads_started":1}
I20260927 00:39:21.145706 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.128922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:59455)
0927 00:39:21.129545 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.129596 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.130701 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.131396 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.131406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.131964 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.132616 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.132629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.133833 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.133836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.135323 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.135334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.136029 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.136033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.136178 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.137085 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.137113 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.139809 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.142023 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.142028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.142566 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.143133 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.143135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.143137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.143210 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.143677 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.143680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.143857 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.144645 (+ 788us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.144796 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":219,"spinlock_wait_cycles":31872,"thread_start_us":660,"threads_started":1}
I20260927 00:39:21.146677 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.128389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46129 (local address 127.0.98.60:42817)
0927 00:39:21.128609 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.128613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.129955 (+ 1342us) server_negotiation.cc:407] Connection header received
0927 00:39:21.131045 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.131049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.131102 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.131177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.132780 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.133890 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135464 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.135714 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.135809 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.139364 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.139368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.139371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.139403 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.141899 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.142454 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.142458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.142460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.142558 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.143098 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.143101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.143103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.143317 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.143400 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.146330 (+ 2930us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.146507 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260927 00:39:21.147339 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42817
I20260927 00:39:21.147353 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36723
I20260927 00:39:21.147354 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:21.147615 7464 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.147615 7466 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.147615 7465 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.148298 7464 heartbeater.cc:511] Master 127.0.98.60:42817 requested a full tablet report, sending...
I20260927 00:39:21.148314 7465 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
I20260927 00:39:21.148314 7466 heartbeater.cc:511] Master 127.0.98.61:36723 requested a full tablet report, sending...
I20260927 00:39:21.149601 6501 ts_manager.cc:194] Re-registered known tserver with Master: 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:21.149936 7284 ts_manager.cc:194] Registered new tserver with Master: 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:21.149937 7103 ts_manager.cc:194] Registered new tserver with Master: 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:21.150319 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59455
W20260927 00:39:21.261653 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:21.261956 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:21.262029 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:21.266283 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:21.266363 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:21.266453 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:21.271929 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35205
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:21.273160 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:21.274209 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:21.276525 6960 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:21.285110 7472 server_base.cc:1034] running on GCE node
W20260927 00:39:21.285091 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.285089 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.285344 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:21.285849 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:21.286614 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:21.287807 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790469561287774 us; error 52 us; skew 500 ppm
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:21.291030 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:39:21.292075 7472 webserver.cc:460] Webserver started at http://127.0.98.2:35205/ using document root <none> and password file <none>
I20260927 00:39:21.292712 7472 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:21.292793 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:21.297387 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:21.299714 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.301175 7472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:21.301290 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "f69de60d50bb446cbb09afa2638bf5bb"
format_stamp: "Formatted at 2026-09-27 00:39:19 on dist-test-slave-xltq"
server_key: "74f2ff4bd4e0a1d230e1e0572275b11e"
server_key_iv: "1b337edb414911917340995ad6189024"
server_key_version: "encryptionkey@0"
I20260927 00:39:21.301867 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:21.312691 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:21.316135 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:21.316349 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:21.316957 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:21.317812 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:21.317859 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.317914 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:21.317941 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.326817 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44685
I20260927 00:39:21.326833 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44685 every 8 connection(s)
I20260927 00:39:21.327874 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:21.334553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7472
I20260927 00:39:21.334724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6831
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:21.341558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37465
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43569
--webserver_port=32863
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:21.345511 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.329681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48303 (local address 127.0.98.62:43569)
0927 00:39:21.329823 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.329828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.331138 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 00:39:21.332166 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.332169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.332253 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.332322 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.333679 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.334501 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.336528 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.336745 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.336842 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.340667 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.340672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.340675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.340705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.342496 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343281 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.343285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.343287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.343382 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343802 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.343806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.343807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.344040 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.344113 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.345227 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.345370 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:39:21.346946 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.330147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:48303)
0927 00:39:21.331024 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.331037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.331955 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.332514 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.332521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.332940 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.333495 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.333516 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.334684 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.334687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.336415 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.336423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.337021 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.337025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.337115 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.338017 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.338049 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.340465 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.342647 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.342654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.342676 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.343160 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.343487 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.343491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.343700 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.344195 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.344232 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.345005 (+ 773us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.345715 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.346107 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":1337,"spinlock_wait_cycles":65536,"thread_start_us":149,"threads_started":1}
I20260927 00:39:21.347689 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.329830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55007 (local address 127.0.98.60:42817)
0927 00:39:21.329956 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.329960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.330970 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 00:39:21.332185 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.332189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.332261 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.332312 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.333836 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.334600 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.336312 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.336693 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.336807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.340500 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.340506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.340509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.340548 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.342688 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343322 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.343326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.343328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.343445 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343868 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.343872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.343874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.344106 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.344183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.347380 (+ 3197us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.347533 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:39:21.348304 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.330148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.98.2:55007)
0927 00:39:21.330778 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.330848 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.331968 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.332516 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.332523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.333581 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.333691 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.333698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.334716 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.334719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.336133 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.336146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.336868 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.336872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.337104 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.338017 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.338044 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.340232 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.342845 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.342849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.342851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.343188 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.343575 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.343578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.343580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.343734 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.344290 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.344293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.344676 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.347914 (+ 3238us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.348074 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":48,"spinlock_wait_cycles":25856,"thread_start_us":647,"threads_started":1}
I20260927 00:39:21.348392 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:21.348632 7602 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.349236 7602 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
I20260927 00:39:21.349938 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.331174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.98.2:37015)
0927 00:39:21.331427 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.331440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.331968 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.332516 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.332523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.332969 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.333496 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.333516 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.340889 (+ 7373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.340892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.342050 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.342064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.342901 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.342906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.343010 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.343528 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.343545 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.345134 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.347598 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.347602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.347604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.348086 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.348752 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.348755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.348756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.348850 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.349393 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.349397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.349500 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.349613 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.349759 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":80,"threads_started":1}
I20260927 00:39:21.349988 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.331549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37015 (local address 127.0.98.61:36723)
0927 00:39:21.331681 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.331686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.331701 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:21.332181 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.332184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.332261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.332312 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.333671 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.334272 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.342234 (+ 7962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.342610 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.342708 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.345289 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.345293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.345295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.345326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.347235 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.348242 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.348247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.348249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.348340 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.348959 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.348963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.348965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.349227 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.349350 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.349687 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.349828 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:39:21.350659 6501 ts_manager.cc:194] Re-registered known tserver with Master: f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685)
I20260927 00:39:21.350589 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42817
I20260927 00:39:21.350951 7601 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.351155 7601 heartbeater.cc:511] Master 127.0.98.60:42817 requested a full tablet report, sending...
I20260927 00:39:21.351434 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48303
I20260927 00:39:21.351773 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36723
I20260927 00:39:21.351862 7603 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.352051 7603 heartbeater.cc:511] Master 127.0.98.61:36723 requested a full tablet report, sending...
I20260927 00:39:21.352037 7284 ts_manager.cc:194] Registered new tserver with Master: f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685)
I20260927 00:39:21.353286 7103 ts_manager.cc:194] Registered new tserver with Master: f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685)
I20260927 00:39:21.408882 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.405522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.0.1:35058)
0927 00:39:21.405659 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.405675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.405803 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.406106 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.406109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.406128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.406224 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.406229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.407022 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.407025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.407762 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.407771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.408464 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.408468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.408557 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.408581 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.408652 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.408739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:39:21.408882 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.405614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.0.98.60:42817)
0927 00:39:21.405729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.405733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.405748 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:21.405920 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.405924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.405980 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.406076 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.406324 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.406903 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.407891 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.408347 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.408441 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.408609 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.408673 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.408738 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:39:21.409643 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 00:39:21.410006 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25135c2052484e7d8704e3ebc7cc8f6c. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:21.410153 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: LEARNER
W20260927 00:39:21.458774 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:21.459129 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:21.459187 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:21.463577 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:21.463649 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:21.463716 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:21.469738 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32863
--tserver_master_addrs=127.0.98.62:43569,127.0.98.61:36723,127.0.98.60:42817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:21.470883 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:21.471958 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:21.482250 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:21.482359 7608 server_base.cc:1034] running on GCE node
W20260927 00:39:21.482277 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:21.482275 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:21.483060 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:21.483793 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:21.485018 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790469561484969 us; error 56 us; skew 500 ppm
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:21.488019 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:39:21.489161 7608 webserver.cc:460] Webserver started at http://127.0.98.3:32863/ using document root <none> and password file <none>
I20260927 00:39:21.489749 7608 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:21.489832 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:21.493970 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:39:21.496279 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.497704 7608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:21.497818 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
format_stamp: "Formatted at 2026-09-27 00:39:20 on dist-test-slave-xltq"
server_key: "55031aae518234f7102b98fe77190ecc"
server_key_iv: "31b8b9049876fb15a63b7fa9b9664766"
server_key_version: "encryptionkey@0"
I20260927 00:39:21.498425 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:21.520287 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:21.523483 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:21.523725 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:21.524415 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:21.525352 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:21.525405 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.525466 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:21.525494 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:21.536662 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37465
I20260927 00:39:21.536710 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37465 every 8 connection(s)
I20260927 00:39:21.537766 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:21.541007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7608
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:39:21.554205 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.543023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.0.98.60:42817)
0927 00:39:21.543282 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.543294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.543721 (+ 427us) server_negotiation.cc:407] Connection header received
0927 00:39:21.543920 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.543924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.543971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.544029 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.545091 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.545807 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.546667 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.546961 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.547063 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.548892 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.548897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.548899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.548926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.552010 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.552492 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.552496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.552498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.552594 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.552893 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.552897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.552899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.553133 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.553257 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.553829 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.553959 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260927 00:39:21.557467 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.544535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.0.98.62:43569)
0927 00:39:21.546798 (+ 2263us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.546803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.547236 (+ 433us) server_negotiation.cc:407] Connection header received
0927 00:39:21.547934 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.547938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.547993 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.548053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.548865 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.549746 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.550971 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.551361 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.551437 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.554028 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.554039 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.554041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.554069 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.555564 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556011 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.556107 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556464 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.556696 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.556799 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.557144 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.557281 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2180,"thread_start_us":80,"threads_started":1}
I20260927 00:39:21.558002 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.545241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.61:36723)
0927 00:39:21.546957 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.546961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.547531 (+ 570us) server_negotiation.cc:407] Connection header received
0927 00:39:21.547575 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.547577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.547622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.547695 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.549922 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.550727 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.551988 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.552551 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.552622 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.554270 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.554275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.554277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.554308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.555937 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556455 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.556549 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556831 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.557046 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.557166 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.557700 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.557822 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648,"thread_start_us":86,"threads_started":1}
I20260927 00:39:21.558485 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.539914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40069 (local address 127.0.98.62:43569)
0927 00:39:21.540025 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.540029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.543444 (+ 3415us) server_negotiation.cc:407] Connection header received
0927 00:39:21.543497 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.543501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.543556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.543650 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.545519 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.546342 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548266 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.548595 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548686 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.552881 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.552885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.552888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.552916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.555418 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556085 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.556166 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556622 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.556626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.556627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.556786 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.556872 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.558203 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.558314 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":896}
I20260927 00:39:21.559367 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.539854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40023 (local address 127.0.98.60:42817)
0927 00:39:21.539997 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.540002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.541412 (+ 1410us) server_negotiation.cc:407] Connection header received
0927 00:39:21.543128 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.543132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.543191 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.543272 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.545512 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.546324 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548328 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.548654 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548717 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.555084 (+ 6367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.555088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.555091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.555115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.556657 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.558122 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.558126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.558127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.558213 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.558469 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.558472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.558474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.558683 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.558759 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.559119 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.559226 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:39:21.559734 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.540476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.98.3:40023)
0927 00:39:21.541227 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.541293 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.542979 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.543625 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.543636 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.544434 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.545338 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.545361 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.546509 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.546513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.548189 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.548196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.548863 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.550638 (+ 1771us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.551131 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.551146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.554920 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.556777 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.556782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.557008 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.558301 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.558303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.558304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.558375 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.558816 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.558818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.558922 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.559042 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.559188 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":2226,"spinlock_wait_cycles":9344,"thread_start_us":156,"threads_started":1}
I20260927 00:39:21.560155 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.540476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:40069)
0927 00:39:21.541560 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.541573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.542620 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.543674 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.543677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.544366 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.545338 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.545361 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.546509 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.546513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.548100 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.548113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.548863 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.548867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.549148 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.549894 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.549917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.552666 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.555559 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.555567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.555588 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.555987 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.556304 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.556307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.556309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.556524 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.556925 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.556933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.557309 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.557528 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.557917 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":332,"spinlock_wait_cycles":25984,"thread_start_us":130,"threads_started":1}
I20260927 00:39:21.561079 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42817
I20260927 00:39:21.561116 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43569
I20260927 00:39:21.561354 7737 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.561354 7738 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.562000 7738 heartbeater.cc:511] Master 127.0.98.62:43569 requested a full tablet report, sending...
I20260927 00:39:21.562835 7737 heartbeater.cc:511] Master 127.0.98.60:42817 requested a full tablet report, sending...
I20260927 00:39:21.563141 6499 ts_manager.cc:194] Re-registered known tserver with Master: 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
I20260927 00:39:21.563597 7284 ts_manager.cc:194] Registered new tserver with Master: 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
Sep 27 00:39:21 dist-test-slave-xltq krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469561, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:39:21.563891 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40069
I20260927 00:39:21.563396 6501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:21.567318 6501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:21.567363 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.542880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52911 (local address 127.0.98.61:36723)
0927 00:39:21.543096 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.543101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.543114 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:21.543151 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.543154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.543256 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.543354 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.548963 (+ 5609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.550381 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.560262 (+ 9881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.560582 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.560660 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.564328 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.564333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.564336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.564366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.565731 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.566155 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.566159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.566161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.566252 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.566567 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.566570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.566572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.566755 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.566842 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.567085 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.567200 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260927 00:39:21.567380 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.541677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.98.3:52911)
0927 00:39:21.541915 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.541928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.542621 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.543779 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.543782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.544399 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:21.546898 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.546908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.557147 (+ 10239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.557150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.557976 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.557986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.560838 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.560842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.560937 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.562581 (+ 1644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:21.562596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:21.564127 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:21.565849 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.565852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.565854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.566050 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.566335 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:21.566338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:21.566340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:21.566429 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:21.566847 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:21.566851 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:21.566938 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.567023 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.567204 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":165,"thread_start_us":71,"threads_started":1}
I20260927 00:39:21.567907 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36723
I20260927 00:39:21.568004 7739 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:21.568260 7739 heartbeater.cc:511] Master 127.0.98.61:36723 requested a full tablet report, sending...
I20260927 00:39:21.568995 7103 ts_manager.cc:194] Registered new tserver with Master: 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
I20260927 00:39:21.570355 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: true } } }
I20260927 00:39:21.570494 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:21.573693 7755 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: attempting to promote NON_VOTER d62f7363a6d84930a5bfef65fc465bc4 to VOTER
I20260927 00:39:21.574057 7755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:21.575037 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 LEARNER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:39:21.575374 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:39:21.575608 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Refusing update from remote peer 25135c2052484e7d8704e3ebc7cc8f6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:39:21.575933 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:39:21.577603 7756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.578040 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.579180 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.579308 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:21.579515 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.579627 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:21.579741 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.579806 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:21.579833 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.580271 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.580399 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:21.581318 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:21.581429 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:21.590708 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.585440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44685 (local address 127.0.0.1:57542)
0927 00:39:21.585614 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.585630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.585707 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.586927 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.586930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.586948 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.587070 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.587077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.588606 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.588610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.589389 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.589397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.590306 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.590310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.590402 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.590434 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.590485 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.590558 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 00:39:21.591567 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.585723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34209 (local address 127.0.0.1:42764)
0927 00:39:21.585942 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.585952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.586033 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.587071 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.587075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.587106 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.587397 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.587403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.588824 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.588827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.590036 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.590057 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.591162 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.591166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.591255 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.591283 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.591344 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.591416 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":218,"spinlock_wait_cycles":2176,"thread_start_us":82,"threads_started":1}
I20260927 00:39:21.591645 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.586405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.98.1:34209)
0927 00:39:21.586695 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.586705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.586723 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:21.586781 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.586785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.586923 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.587255 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.587503 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.588695 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.590182 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.591036 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.591158 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.591327 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.591397 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.591504 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260927 00:39:21.591645 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.585612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.2:44685)
0927 00:39:21.586121 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.586133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.586158 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:39:21.586250 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.586256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.586786 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.587374 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.587629 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.588496 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.589513 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.590173 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.591229 (+ 1056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.591354 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.591396 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.591499 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":138,"threads_started":1}
I20260927 00:39:21.591938 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.586021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37465 (local address 127.0.0.1:46552)
0927 00:39:21.586241 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.586252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.586330 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.587071 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.587075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.587106 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.587244 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.587251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.589357 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.589361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.590487 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.590496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.591583 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.591586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.591670 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.591695 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.591747 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.591824 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":56,"spinlock_wait_cycles":8448,"thread_start_us":104,"threads_started":1}
I20260927 00:39:21.595080 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 33a728ac357942149481be23d08ae00a (DEFAULT_TABLE table=test-table [id=4f62f342dbd748a19a41805ff958b117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:21.595079 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 33a728ac357942149481be23d08ae00a (DEFAULT_TABLE table=test-table [id=4f62f342dbd748a19a41805ff958b117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:21.596436 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a728ac357942149481be23d08ae00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:21.596464 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a728ac357942149481be23d08ae00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:21.599731 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.586202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.0.98.3:37465)
0927 00:39:21.586603 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.586614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.586636 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:39:21.586708 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.586713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.586915 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.587523 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.587653 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.589239 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.590623 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.591430 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.592088 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.592282 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.592335 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.599560 (+ 7225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":96,"threads_started":1}
I20260927 00:39:21.602662 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 33a728ac357942149481be23d08ae00a (DEFAULT_TABLE table=test-table [id=4f62f342dbd748a19a41805ff958b117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:21.603673 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a728ac357942149481be23d08ae00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:21.604899 7770 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Bootstrap starting.
I20260927 00:39:21.605965 7771 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Bootstrap starting.
I20260927 00:39:21.607707 7770 tablet_bootstrap.cc:654] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:21.607826 7771 tablet_bootstrap.cc:654] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:21.608554 7771 log.cc:824] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:21.608594 7770 log.cc:824] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:21.610569 7771 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: No bootstrap required, opened a new log
I20260927 00:39:21.610569 7770 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: No bootstrap required, opened a new log
I20260927 00:39:21.610770 7771 ts_tablet_manager.cc:1397] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:39:21.610770 7770 ts_tablet_manager.cc:1397] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:21.612100 7774 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Bootstrap starting.
I20260927 00:39:21.614533 7774 tablet_bootstrap.cc:654] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:21.615264 7774 log.cc:824] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:21.615327 7771 raft_consensus.cc:348] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.615327 7770 raft_consensus.cc:348] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.615617 7770 raft_consensus.cc:374] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:21.615617 7771 raft_consensus.cc:374] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:21.615674 7771 raft_consensus.cc:729] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69de60d50bb446cbb09afa2638bf5bb, State: Initialized, Role: FOLLOWER
I20260927 00:39:21.615674 7770 raft_consensus.cc:729] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590e1ec5c8b645d78fbbcb9b42f5df2b, State: Initialized, Role: FOLLOWER
I20260927 00:39:21.616147 7770 consensus_queue.cc:260] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.616147 7771 consensus_queue.cc:260] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.617208 7770 ts_tablet_manager.cc:1428] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:39:21.617208 7771 ts_tablet_manager.cc:1428] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 00:39:21.617425 7774 tablet_bootstrap.cc:492] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: No bootstrap required, opened a new log
I20260927 00:39:21.617437 7465 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:21.617623 7774 ts_tablet_manager.cc:1397] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:21.619513 7602 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:21.622809 7774 raft_consensus.cc:348] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.623098 7774 raft_consensus.cc:374] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:21.623159 7774 raft_consensus.cc:729] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c56ed0e50043768cb9c0c1d059ee2c, State: Initialized, Role: FOLLOWER
I20260927 00:39:21.623698 7774 consensus_queue.cc:260] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.624598 7738 heartbeater.cc:503] Master 127.0.98.62:43569 was elected leader, sending a full tablet report...
I20260927 00:39:21.624956 7774 ts_tablet_manager.cc:1428] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
W20260927 00:39:21.627990 7467 tablet.cc:2369] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:39:21.775941 7776 raft_consensus.cc:482] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:21.776225 7776 raft_consensus.cc:504] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.777482 7776 leader_election.cc:288] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685), 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
I20260927 00:39:21.781620 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.777779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44685 (local address 127.0.98.1:58937)
0927 00:39:21.778185 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.778199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.778304 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.778565 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.778568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.778584 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.778678 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.778683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.779408 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.779412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.780451 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.780461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.781113 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.781117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.781265 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.781285 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.781350 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.781458 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":97,"thread_start_us":81,"threads_started":1}
I20260927 00:39:21.781599 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.777874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58937 (local address 127.0.98.2:44685)
0927 00:39:21.778037 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.778042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.778286 (+ 244us) server_negotiation.cc:407] Connection header received
0927 00:39:21.778421 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.778425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.778475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.778542 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.778774 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.779296 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.780576 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.780993 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.781093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.781162 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.781378 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.781468 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:21.781599 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.778025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51489 (local address 127.0.98.3:37465)
0927 00:39:21.778136 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.778141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.778341 (+ 200us) server_negotiation.cc:407] Connection header received
0927 00:39:21.778484 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.778487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.778542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.778607 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:21.778819 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.779332 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.780569 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.780984 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.781094 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.781161 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.781374 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.781449 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:39:21.781641 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.777779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37465 (local address 127.0.98.1:51489)
0927 00:39:21.778253 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:21.778267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:21.778387 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:21.778625 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:21.778628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:21.778643 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:21.778730 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.778735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.779432 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.779436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.780451 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:21.780461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.781113 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.781117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:21.781269 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.781288 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:39:21.781352 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:39:21.781452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":123,"thread_start_us":70,"threads_started":1}
I20260927 00:39:21.782644 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a728ac357942149481be23d08ae00a" candidate_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69de60d50bb446cbb09afa2638bf5bb" is_pre_election: true
I20260927 00:39:21.782641 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a728ac357942149481be23d08ae00a" candidate_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" is_pre_election: true
I20260927 00:39:21.782935 7687 raft_consensus.cc:2454] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590e1ec5c8b645d78fbbcb9b42f5df2b in term 0.
I20260927 00:39:21.782941 7555 raft_consensus.cc:2454] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590e1ec5c8b645d78fbbcb9b42f5df2b in term 0.
I20260927 00:39:21.783376 7354 leader_election.cc:302] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c56ed0e50043768cb9c0c1d059ee2c, 590e1ec5c8b645d78fbbcb9b42f5df2b; no voters:
I20260927 00:39:21.783655 7776 raft_consensus.cc:2781] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:21.783738 7776 raft_consensus.cc:482] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:21.783790 7776 raft_consensus.cc:3037] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:21.784860 7776 raft_consensus.cc:504] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
I20260927 00:39:21.785238 7776 leader_election.cc:288] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [CANDIDATE]: Term 1 election: Requested vote from peers f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685), 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465)
I20260927 00:39:21.785596 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a728ac357942149481be23d08ae00a" candidate_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69de60d50bb446cbb09afa2638bf5bb"
I20260927 00:39:21.785633 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a728ac357942149481be23d08ae00a" candidate_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
I20260927 00:39:21.785746 7555 raft_consensus.cc:3037] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:21.785746 7687 raft_consensus.cc:3037] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:21.787011 7555 raft_consensus.cc:2454] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590e1ec5c8b645d78fbbcb9b42f5df2b in term 1.
I20260927 00:39:21.787022 7687 raft_consensus.cc:2454] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590e1ec5c8b645d78fbbcb9b42f5df2b in term 1.
I20260927 00:39:21.787351 7354 leader_election.cc:302] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c56ed0e50043768cb9c0c1d059ee2c, 590e1ec5c8b645d78fbbcb9b42f5df2b; no voters:
I20260927 00:39:21.787599 7776 raft_consensus.cc:2781] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:21.787801 7776 raft_consensus.cc:686] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 LEADER]: Becoming Leader. State: Replica: 590e1ec5c8b645d78fbbcb9b42f5df2b, State: Running, Role: LEADER
I20260927 00:39:21.788306 7776 consensus_queue.cc:237] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } }
W20260927 00:39:21.789865 7740 tablet.cc:2369] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:39:21.792052 6499 catalog_manager.cc:5495] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:21.825439 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.821826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.98.1:34209)
0927 00:39:21.821968 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.821972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.821987 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:21.822140 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.822143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.822200 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.822297 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:21.822556 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.823121 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.823986 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.824194 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.824306 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.824619 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:21.825058 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:21.825166 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.825229 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.825299 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 00:39:21.829839 7604 tablet.cc:2369] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:39:21.831516 7687 raft_consensus.cc:1261] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Refusing update from remote peer 590e1ec5c8b645d78fbbcb9b42f5df2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:21.831516 7555 raft_consensus.cc:1261] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Refusing update from remote peer 590e1ec5c8b645d78fbbcb9b42f5df2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:21.832409 7776 consensus_queue.cc:1035] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:21.832624 7781 consensus_queue.cc:1035] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:21.841905 7784 mvcc.cc:205] Tried to move back new op lower bound from 7333763325253918720 to 7333763325087649792. Current Snapshot: MvccSnapshot[applied={T|T < 7333763325253918720}]
I20260927 00:39:21.842316 7785 mvcc.cc:205] Tried to move back new op lower bound from 7333763325253918720 to 7333763325087649792. Current Snapshot: MvccSnapshot[applied={T|T < 7333763325253918720}]
I20260927 00:39:21.846071 7786 mvcc.cc:205] Tried to move back new op lower bound from 7333763325253918720 to 7333763325087649792. Current Snapshot: MvccSnapshot[applied={T|T < 7333763325253918720 or (T in {7333763325253918720})}]
I20260927 00:39:21.851248 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:21.851425 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:21.855643 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25135c2052484e7d8704e3ebc7cc8f6c: Sending DeleteTablet for 3 replicas of tablet 33a728ac357942149481be23d08ae00a
I20260927 00:39:21.856642 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a728ac357942149481be23d08ae00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42764
I20260927 00:39:21.856642 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a728ac357942149481be23d08ae00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46552
I20260927 00:39:21.857189 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a728ac357942149481be23d08ae00a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57542
I20260927 00:39:21.857478 7795 tablet_replica.cc:330] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: stopping tablet replica
I20260927 00:39:21.857566 7796 tablet_replica.cc:330] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: stopping tablet replica
I20260927 00:39:21.857677 7794 tablet_replica.cc:330] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: stopping tablet replica
I20260927 00:39:21.857959 7795 raft_consensus.cc:2229] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:21.857954 7796 raft_consensus.cc:2229] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:21.858312 7796 raft_consensus.cc:2258] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:21.858312 7794 raft_consensus.cc:2229] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:21.858894 7794 raft_consensus.cc:2258] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:21.859263 7795 raft_consensus.cc:2258] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:21.860008 7796 ts_tablet_manager.cc:1905] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:21.860741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6469
I20260927 00:39:21.861984 7795 ts_tablet_manager.cc:1905] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:21.862521 7794 ts_tablet_manager.cc:1905] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:21.863935 7796 ts_tablet_manager.cc:1918] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:21.864042 7796 log.cc:1196] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/33a728ac357942149481be23d08ae00a
I20260927 00:39:21.864370 7796 ts_tablet_manager.cc:1939] T 33a728ac357942149481be23d08ae00a P f69de60d50bb446cbb09afa2638bf5bb: Deleting consensus metadata
I20260927 00:39:21.866550 7794 ts_tablet_manager.cc:1918] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:21.866653 7794 log.cc:1196] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/33a728ac357942149481be23d08ae00a
I20260927 00:39:21.866917 7794 ts_tablet_manager.cc:1939] T 33a728ac357942149481be23d08ae00a P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting consensus metadata
W20260927 00:39:21.868112 7627 connection.cc:441] server connection from 127.0.0.1:46552 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46552 (request call id 1) could send its response
W20260927 00:39:21.868693 7488 connection.cc:535] server connection from 127.0.0.1:57542 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:39:21.869225 7795 ts_tablet_manager.cc:1918] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:21.869314 7795 log.cc:1196] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/33a728ac357942149481be23d08ae00a
I20260927 00:39:21.869560 7795 ts_tablet_manager.cc:1939] T 33a728ac357942149481be23d08ae00a P 590e1ec5c8b645d78fbbcb9b42f5df2b: Deleting consensus metadata
W20260927 00:39:21.870440 7352 connection.cc:441] server connection from 127.0.0.1:42764 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42764 (request call id 1) could send its response
I20260927 00:39:21.882081 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.871801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.60:42817)
0927 00:39:21.871935 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.871940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.872623 (+ 683us) server_negotiation.cc:407] Connection header received
0927 00:39:21.872792 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.872795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.872846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.872932 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.873770 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.874539 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.875655 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.876026 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.876099 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.877725 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.877729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.877731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.877760 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.880137 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.880728 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.880732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.880734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.880822 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.881115 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.881118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.881119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.881339 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.881440 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.881763 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.881914 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:39:21.882236 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.872454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.0.98.61:36723)
0927 00:39:21.872566 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.872570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.872948 (+ 378us) server_negotiation.cc:407] Connection header received
0927 00:39:21.873102 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.873105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.873153 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.873202 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.874616 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.875391 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.876465 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.876770 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.876884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.878915 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.878919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.878921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.878946 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.880648 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.881039 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.881042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.881044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.881121 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.881401 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.881405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.881406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.881602 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.881693 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.881995 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.882108 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:39:21.882594 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.859797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.98.61:36723)
0927 00:39:21.859977 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.859982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.871763 (+ 11781us) server_negotiation.cc:407] Connection header received
0927 00:39:21.872097 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.872101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.872154 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.872228 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.873301 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.874122 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.875403 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.875736 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.875825 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.878373 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.878377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.878380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.878407 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.880408 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.880869 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.880873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.880875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.880964 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.881452 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.881456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.881458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.881660 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.881772 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.882347 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.882451 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:39:21.891573 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:21.860187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.0.98.60:42817)
0927 00:39:21.860317 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:21.860321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:21.883128 (+ 22807us) server_negotiation.cc:407] Connection header received
0927 00:39:21.884064 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:21.884067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:21.884116 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:21.884191 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:21.884962 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.885513 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.886635 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:21.886940 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:21.887034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:21.888556 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:21.888560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:21.888563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:21.888590 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:21.890041 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.890427 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.890431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.890432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.890505 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:21.890755 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:21.890759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:21.890760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:21.890945 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:21.891060 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:21.891294 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 00:39:21.891409 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:39:22.845600 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:22.844916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:40329)
0927 00:39:22.845295 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:22.845448 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":86,"threads_started":1}
W20260927 00:39:22.846084 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43569 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:22.848953 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:22.848479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:58533)
0927 00:39:22.848708 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:22.848832 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
W20260927 00:39:22.849438 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43569 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:22.850109 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:22.849366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:40813)
0927 00:39:22.849883 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:22.849992 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":94,"threads_started":1}
W20260927 00:39:22.850512 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43569 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.356566 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25135c2052484e7d8704e3ebc7cc8f6c)
I20260927 00:39:23.356778 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:23.357110 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25135c2052484e7d8704e3ebc7cc8f6c)
I20260927 00:39:23.357340 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:23.357990 7808 leader_election.cc:288] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), b74d423ad6374397bca360fb094c16bd (127.0.98.61:36723)
I20260927 00:39:23.358670 7809 leader_election.cc:288] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60:42817)
I20260927 00:39:23.359517 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.358247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60862)
0927 00:39:23.359278 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.359372 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":57,"threads_started":1}
W20260927 00:39:23.359890 7269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.360704 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.358801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60874)
0927 00:39:23.360493 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.360585 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":86,"threads_started":1}
W20260927 00:39:23.361089 7087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.361214 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.360904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60904)
0927 00:39:23.361047 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.361120 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 00:39:23.361310 7269 leader_election.cc:334] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.362253 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.361973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60908)
0927 00:39:23.362106 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.362165 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 00:39:23.362396 7087 leader_election.cc:334] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.364076 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.359054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.98.61:36723)
0927 00:39:23.360197 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:23.360221 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:23.360237 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:39:23.360285 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:23.360287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:23.360362 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:23.360462 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:23.360817 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.361631 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.362991 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.363456 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.363570 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:23.363646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:23.363826 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 00:39:23.363924 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"spinlock_wait_cycles":6528,"thread_start_us":72,"threads_started":1}
I20260927 00:39:23.364073 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.358509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36723 (local address 127.0.0.1:37082)
0927 00:39:23.359671 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.359686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:23.359825 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:23.360508 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:23.360512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:23.360535 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:23.360674 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:23.360682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.361742 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.361745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:23.362858 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:23.362869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.363593 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.363597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:23.363707 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:23.363729 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:39:23.363802 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:39:23.363917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":59,"threads_started":1}
I20260927 00:39:23.364282 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.359521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.98.60:42817)
0927 00:39:23.360082 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:23.360086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:23.360583 (+ 497us) server_negotiation.cc:407] Connection header received
0927 00:39:23.360639 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:23.360642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:23.360690 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:23.361086 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:23.361191 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.361919 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.363209 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.363656 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.363748 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:23.363815 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:23.363969 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 00:39:23.364053 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":93,"threads_started":1}
I20260927 00:39:23.364284 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.358961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42817 (local address 127.0.0.1:35104)
0927 00:39:23.360179 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.360194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:23.360359 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:23.360832 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:23.360835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:23.360855 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:23.360980 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:23.360987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.362023 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.362026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:23.363092 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:23.363102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:23.363763 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:23.363767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:23.363855 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:23.363880 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:39:23.363944 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:39:23.364037 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"spinlock_wait_cycles":768,"thread_start_us":76,"threads_started":1}
I20260927 00:39:23.365016 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d62f7363a6d84930a5bfef65fc465bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b74d423ad6374397bca360fb094c16bd" is_pre_election: true
I20260927 00:39:23.365242 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b74d423ad6374397bca360fb094c16bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d62f7363a6d84930a5bfef65fc465bc4" is_pre_election: true
I20260927 00:39:23.365307 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62f7363a6d84930a5bfef65fc465bc4 in term 1.
I20260927 00:39:23.365499 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b74d423ad6374397bca360fb094c16bd in term 1.
I20260927 00:39:23.365722 7269 leader_election.cc:302] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b74d423ad6374397bca360fb094c16bd, d62f7363a6d84930a5bfef65fc465bc4; no voters: 25135c2052484e7d8704e3ebc7cc8f6c
I20260927 00:39:23.365900 7090 leader_election.cc:302] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b74d423ad6374397bca360fb094c16bd, d62f7363a6d84930a5bfef65fc465bc4; no voters: 25135c2052484e7d8704e3ebc7cc8f6c
I20260927 00:39:23.366021 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:39:23.366089 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25135c2052484e7d8704e3ebc7cc8f6c)
I20260927 00:39:23.366135 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:39:23.366263 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:39:23.366325 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25135c2052484e7d8704e3ebc7cc8f6c)
I20260927 00:39:23.366371 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:39:23.367226 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:23.367589 7808 leader_election.cc:288] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 election: Requested vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), b74d423ad6374397bca360fb094c16bd (127.0.98.61:36723)
I20260927 00:39:23.367746 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:23.368088 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.367820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60924)
0927 00:39:23.367956 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.368021 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:39:23.368281 7809 leader_election.cc:288] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 election: Requested vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), d62f7363a6d84930a5bfef65fc465bc4 (127.0.98.60:42817)
I20260927 00:39:23.368345 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d62f7363a6d84930a5bfef65fc465bc4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b74d423ad6374397bca360fb094c16bd"
I20260927 00:39:23.368543 7113 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d62f7363a6d84930a5bfef65fc465bc4 in current term 2: Already voted for candidate b74d423ad6374397bca360fb094c16bd in this term.
I20260927 00:39:23.368639 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b74d423ad6374397bca360fb094c16bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d62f7363a6d84930a5bfef65fc465bc4"
I20260927 00:39:23.368774 7295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b74d423ad6374397bca360fb094c16bd in current term 2: Already voted for candidate d62f7363a6d84930a5bfef65fc465bc4 in this term.
I20260927 00:39:23.369105 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.368854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60936)
0927 00:39:23.368982 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.369050 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:39:23.369222 7269 leader_election.cc:334] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.369309 7269 leader_election.cc:302] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d62f7363a6d84930a5bfef65fc465bc4; no voters: 25135c2052484e7d8704e3ebc7cc8f6c, b74d423ad6374397bca360fb094c16bd
I20260927 00:39:23.369442 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.368370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60932)
0927 00:39:23.369243 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.369304 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783}
I20260927 00:39:23.369457 7808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:39:23.370136 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.369915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:60942)
0927 00:39:23.370032 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.370081 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:39:23.370262 7087 leader_election.cc:334] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:23.370332 7087 leader_election.cc:302] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b74d423ad6374397bca360fb094c16bd; no voters: 25135c2052484e7d8704e3ebc7cc8f6c, d62f7363a6d84930a5bfef65fc465bc4
I20260927 00:39:23.370507 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:39:23.847263 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.846666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:45125)
0927 00:39:23.847071 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.847141 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":101,"threads_started":1}
I20260927 00:39:23.850857 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.850190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:38197)
0927 00:39:23.850613 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.850700 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":186,"threads_started":1}
I20260927 00:39:23.851804 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:23.851254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:45019)
0927 00:39:23.851620 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:23.851695 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260927 00:39:24.849223 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.848583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:59511)
0927 00:39:24.848971 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.849055 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":107,"threads_started":1}
I20260927 00:39:24.852350 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.851756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:42765)
0927 00:39:24.852094 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.852195 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260927 00:39:24.853149 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.852696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:41295)
0927 00:39:24.852961 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.853032 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260927 00:39:24.943802 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:24.944006 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:24.944581 7824 leader_election.cc:288] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), b74d423ad6374397bca360fb094c16bd (127.0.98.61:36723)
I20260927 00:39:24.945453 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.944888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58874)
0927 00:39:24.945243 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.945335 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260927 00:39:24.945920 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d62f7363a6d84930a5bfef65fc465bc4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b74d423ad6374397bca360fb094c16bd" is_pre_election: true
I20260927 00:39:24.946166 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62f7363a6d84930a5bfef65fc465bc4 in term 2.
I20260927 00:39:24.946377 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.946176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58882)
0927 00:39:24.946267 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.946317 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 00:39:24.946547 7269 leader_election.cc:302] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b74d423ad6374397bca360fb094c16bd, d62f7363a6d84930a5bfef65fc465bc4; no voters:
W20260927 00:39:24.946830 7269 leader_election.cc:334] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:24.946882 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 00:39:24.946938 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:24.946964 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:39:24.948297 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:24.948697 7824 leader_election.cc:288] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 election: Requested vote from peers 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569), b74d423ad6374397bca360fb094c16bd (127.0.98.61:36723)
I20260927 00:39:24.949137 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.948887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58886)
0927 00:39:24.948993 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.949057 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:39:24.949553 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d62f7363a6d84930a5bfef65fc465bc4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b74d423ad6374397bca360fb094c16bd"
I20260927 00:39:24.949726 7113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:39:24.949857 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:24.949637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58892)
0927 00:39:24.949728 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:24.949798 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:39:24.949958 7269 leader_election.cc:334] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
I20260927 00:39:24.951046 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62f7363a6d84930a5bfef65fc465bc4 in term 3.
I20260927 00:39:24.951443 7269 leader_election.cc:302] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b74d423ad6374397bca360fb094c16bd, d62f7363a6d84930a5bfef65fc465bc4; no voters: 25135c2052484e7d8704e3ebc7cc8f6c
I20260927 00:39:24.951668 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:39:24.951918 7824 raft_consensus.cc:686] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [term 3 LEADER]: Becoming Leader. State: Replica: d62f7363a6d84930a5bfef65fc465bc4, State: Running, Role: LEADER
I20260927 00:39:24.952236 7824 consensus_queue.cc:237] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } }
I20260927 00:39:24.954084 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d62f7363a6d84930a5bfef65fc465bc4. Latest consensus state: current_term: 3 leader_uuid: "d62f7363a6d84930a5bfef65fc465bc4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:24.954257 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:24.954720 7829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:24.957448 7829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:24.958132 7829 catalog_manager.cc:1259] Loaded cluster ID: 175a739dde8c46b692137ba1d9430c16
I20260927 00:39:24.958199 7829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:24.958999 7829 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:24.959470 7829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Loaded TSK: 0
I20260927 00:39:24.960459 7829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:25.050899 7284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:25.054510 7284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:25.058491 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.058073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58934)
0927 00:39:25.058254 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.058410 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 00:39:25.058545 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [term 3 FOLLOWER]: Refusing update from remote peer d62f7363a6d84930a5bfef65fc465bc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 00:39:25.059355 7824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 00:39:25.059458 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.059117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.0.1:58936)
0927 00:39:25.059254 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.059331 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 00:39:25.059633 7269 consensus_peers.cc:487] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 -> Peer 25135c2052484e7d8704e3ebc7cc8f6c (127.0.98.62:43569): Couldn't send request to peer 25135c2052484e7d8704e3ebc7cc8f6c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:39:25.061111 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d62f7363a6d84930a5bfef65fc465bc4. Latest consensus state: current_term: 3 leader_uuid: "d62f7363a6d84930a5bfef65fc465bc4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:25.061309 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:25.062592 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d62f7363a6d84930a5bfef65fc465bc4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:25.062743 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:25.062989 7832 mvcc.cc:205] Tried to move back new op lower bound from 7333763338472583168 to 7333763338047229952. Current Snapshot: MvccSnapshot[applied={T|T < 7333763338472583168}]
I20260927 00:39:25.063169 7831 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d62f7363a6d84930a5bfef65fc465bc4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "25135c2052484e7d8704e3ebc7cc8f6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43569 } } peers { permanent_uuid: "b74d423ad6374397bca360fb094c16bd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36723 } attrs { promote: false } } peers { permanent_uuid: "d62f7363a6d84930a5bfef65fc465bc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42817 } attrs { promote: false } } }
I20260927 00:39:25.063272 7831 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74d423ad6374397bca360fb094c16bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:39:25.076856 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.071485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37465 (local address 127.0.0.1:53952)
0927 00:39:25.071667 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.071687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:25.071813 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:25.072428 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:25.072431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:25.072454 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:25.072606 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.072612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.073937 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.073941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.075253 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.075265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.076400 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.076404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.076504 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.076527 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:39:25.076580 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:39:25.076679 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":86}
I20260927 00:39:25.077212 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.072116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.0.98.1:34209)
0927 00:39:25.072925 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.072930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.072944 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:25.072998 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.073001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.073049 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.073142 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:25.073417 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.074139 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.075530 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.076232 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.076348 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.076420 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.076981 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.077073 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":102,"threads_started":1}
I20260927 00:39:25.076910 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.071986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34209 (local address 127.0.0.1:55336)
0927 00:39:25.072413 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.072427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:25.072502 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:25.073170 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:25.073173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:25.073192 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:25.073289 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.073295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.074255 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.074258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.075392 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.075402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.076472 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.076476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.076572 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.076592 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:25.076645 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 00:39:25.076710 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":162,"threads_started":1}
I20260927 00:39:25.077440 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.071719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.98.3:37465)
0927 00:39:25.072079 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.072084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.072106 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:39:25.072178 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.072181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.072274 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.072710 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:25.072839 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.073832 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.075555 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.076250 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.077099 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.077181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.077217 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.077304 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260927 00:39:25.078020 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 38da0334af12471ba2c426fe33399059 (DEFAULT_TABLE table=test-table [id=004ece52da0a423f8b163689a8f3f3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:25.078001 7665 tablet_service.cc:1467] Processing CreateTablet for tablet 38da0334af12471ba2c426fe33399059 (DEFAULT_TABLE table=test-table [id=004ece52da0a423f8b163689a8f3f3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:25.078303 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38da0334af12471ba2c426fe33399059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:25.078303 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38da0334af12471ba2c426fe33399059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:25.078585 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.072122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44685 (local address 127.0.0.1:59518)
0927 00:39:25.072652 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.072665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:25.072748 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:25.073699 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:25.073702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:25.073720 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:25.073825 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.073831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.075963 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.075967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.077024 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.077032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.078137 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.078140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.078234 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.078260 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:39:25.078326 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:39:25.078430 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20260927 00:39:25.079041 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.072393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.0.98.2:44685)
0927 00:39:25.073366 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.073371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.073384 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:25.073432 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.073434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.073490 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.073601 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:25.075162 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.075846 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.077177 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.077885 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.077991 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.078079 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.078789 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.078902 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":65,"threads_started":1}
I20260927 00:39:25.079620 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 38da0334af12471ba2c426fe33399059 (DEFAULT_TABLE table=test-table [id=004ece52da0a423f8b163689a8f3f3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:25.079957 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38da0334af12471ba2c426fe33399059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:25.081144 7841 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: Bootstrap starting.
I20260927 00:39:25.081497 7842 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Bootstrap starting.
I20260927 00:39:25.082665 7842 tablet_bootstrap.cc:654] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:25.084316 7842 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: No bootstrap required, opened a new log
I20260927 00:39:25.084442 7842 ts_tablet_manager.cc:1397] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:25.084475 7841 tablet_bootstrap.cc:654] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:25.084646 7843 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Bootstrap starting.
I20260927 00:39:25.084945 7842 raft_consensus.cc:348] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.085054 7842 raft_consensus.cc:374] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:25.085084 7842 raft_consensus.cc:729] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c56ed0e50043768cb9c0c1d059ee2c, State: Initialized, Role: FOLLOWER
I20260927 00:39:25.085208 7842 consensus_queue.cc:260] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.085618 7842 ts_tablet_manager.cc:1428] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:25.085851 7843 tablet_bootstrap.cc:654] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:25.086690 7841 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: No bootstrap required, opened a new log
I20260927 00:39:25.086819 7841 ts_tablet_manager.cc:1397] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:39:25.087316 7841 raft_consensus.cc:348] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.087469 7841 raft_consensus.cc:374] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:25.087502 7841 raft_consensus.cc:729] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590e1ec5c8b645d78fbbcb9b42f5df2b, State: Initialized, Role: FOLLOWER
I20260927 00:39:25.087647 7841 consensus_queue.cc:260] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.088105 7841 ts_tablet_manager.cc:1428] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:39:25.089186 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.088875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:37639)
0927 00:39:25.089027 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.089097 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 00:39:25.089882 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.088897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:41049)
0927 00:39:25.089717 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.089789 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725}
I20260927 00:39:25.090238 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.089997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:33007)
0927 00:39:25.090131 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.090173 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 00:39:25.090517 7737 heartbeater.cc:503] Master 127.0.98.60:42817 was elected leader, sending a full tablet report...
I20260927 00:39:25.091367 7464 heartbeater.cc:503] Master 127.0.98.60:42817 was elected leader, sending a full tablet report...
I20260927 00:39:25.093590 7843 tablet_bootstrap.cc:492] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: No bootstrap required, opened a new log
I20260927 00:39:25.093739 7843 ts_tablet_manager.cc:1397] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 00:39:25.094220 7843 raft_consensus.cc:348] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.094341 7843 raft_consensus.cc:374] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:25.094362 7843 raft_consensus.cc:729] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69de60d50bb446cbb09afa2638bf5bb, State: Initialized, Role: FOLLOWER
I20260927 00:39:25.094492 7843 consensus_queue.cc:260] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.094880 7843 ts_tablet_manager.cc:1428] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:25.095693 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.095393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:40343)
0927 00:39:25.095526 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.095594 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:39:25.096762 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.096351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:34147)
0927 00:39:25.096610 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.096666 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260927 00:39:25.096771 7601 heartbeater.cc:503] Master 127.0.98.60:42817 was elected leader, sending a full tablet report...
I20260927 00:39:25.153249 7849 raft_consensus.cc:482] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:25.153484 7849 raft_consensus.cc:504] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.154976 7849 leader_election.cc:288] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465), 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:25.159544 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.155656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42819 (local address 127.0.98.1:34209)
0927 00:39:25.155770 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.155773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.156145 (+ 372us) server_negotiation.cc:407] Connection header received
0927 00:39:25.156233 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.156236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.156288 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.156370 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:25.156679 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.157255 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.158450 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.158930 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.159048 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.159116 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.159343 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.159418 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:25.159566 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.155352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51575 (local address 127.0.98.3:37465)
0927 00:39:25.155489 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.155494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.155515 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:39:25.155713 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.155717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.155784 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.155864 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:25.156318 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.156989 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.158266 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.158773 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.158869 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.158941 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.159300 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.159399 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:25.159581 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.155550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34209 (local address 127.0.98.2:42819)
0927 00:39:25.156020 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.156034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:25.156123 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:25.156391 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:25.156394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:25.156410 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:25.156555 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.156563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.157407 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.157411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.158305 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.158317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.159077 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.159080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.159201 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.159227 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:39:25.159312 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:39:25.159403 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":3,"thread_start_us":159,"threads_started":1}
I20260927 00:39:25.159544 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.155248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37465 (local address 127.0.98.2:51575)
0927 00:39:25.155395 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.155413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:25.155533 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:25.155916 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:25.155920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:25.155955 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:25.156120 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.156130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.157116 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.157120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.158099 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:25.158110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.158943 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.158947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:25.159155 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.159183 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:39:25.159251 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:39:25.159339 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":35}
I20260927 00:39:25.160179 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38da0334af12471ba2c426fe33399059" candidate_uuid: "f69de60d50bb446cbb09afa2638bf5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" is_pre_election: true
I20260927 00:39:25.160411 7687 raft_consensus.cc:2454] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69de60d50bb446cbb09afa2638bf5bb in term 0.
I20260927 00:39:25.160552 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38da0334af12471ba2c426fe33399059" candidate_uuid: "f69de60d50bb446cbb09afa2638bf5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" is_pre_election: true
I20260927 00:39:25.160900 7418 raft_consensus.cc:2454] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69de60d50bb446cbb09afa2638bf5bb in term 0.
I20260927 00:39:25.160899 7491 leader_election.cc:302] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c56ed0e50043768cb9c0c1d059ee2c, f69de60d50bb446cbb09afa2638bf5bb; no voters:
I20260927 00:39:25.161288 7849 raft_consensus.cc:2781] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:25.161360 7849 raft_consensus.cc:482] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:25.161399 7849 raft_consensus.cc:3037] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:25.162521 7849 raft_consensus.cc:504] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.162919 7849 leader_election.cc:288] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [CANDIDATE]: Term 1 election: Requested vote from peers 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465), 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209)
I20260927 00:39:25.163442 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38da0334af12471ba2c426fe33399059" candidate_uuid: "f69de60d50bb446cbb09afa2638bf5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c56ed0e50043768cb9c0c1d059ee2c"
I20260927 00:39:25.163592 7687 raft_consensus.cc:3037] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:25.163782 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38da0334af12471ba2c426fe33399059" candidate_uuid: "f69de60d50bb446cbb09afa2638bf5bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b"
I20260927 00:39:25.163973 7418 raft_consensus.cc:3037] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:25.164569 7687 raft_consensus.cc:2454] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69de60d50bb446cbb09afa2638bf5bb in term 1.
I20260927 00:39:25.164943 7491 leader_election.cc:302] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c56ed0e50043768cb9c0c1d059ee2c, f69de60d50bb446cbb09afa2638bf5bb; no voters:
I20260927 00:39:25.165208 7849 raft_consensus.cc:2781] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:25.165501 7849 raft_consensus.cc:686] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 1 LEADER]: Becoming Leader. State: Replica: f69de60d50bb446cbb09afa2638bf5bb, State: Running, Role: LEADER
I20260927 00:39:25.165593 7418 raft_consensus.cc:2454] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69de60d50bb446cbb09afa2638bf5bb in term 1.
I20260927 00:39:25.165869 7849 consensus_queue.cc:237] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } }
I20260927 00:39:25.166505 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:60475)
0927 00:39:25.166332 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.166410 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 00:39:25.169320 7283 catalog_manager.cc:5495] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb reported cstate change: term changed from 0 to 1, leader changed from <none> to f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f69de60d50bb446cbb09afa2638bf5bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f69de60d50bb446cbb09afa2638bf5bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:25.209962 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.205383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.0.98.2:44685)
0927 00:39:25.205537 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:25.205542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:25.205562 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:39:25.205772 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:25.205775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:25.205847 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:25.205958 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:25.206286 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.206929 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.207953 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:25.208248 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:25.208345 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:25.208729 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:25.209461 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:25.209586 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:25.209687 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 00:39:25.209765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:39:25.213886 7687 raft_consensus.cc:1261] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Refusing update from remote peer f69de60d50bb446cbb09afa2638bf5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:25.214286 7418 raft_consensus.cc:1261] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Refusing update from remote peer f69de60d50bb446cbb09afa2638bf5bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:25.214735 7851 consensus_queue.cc:1035] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c56ed0e50043768cb9c0c1d059ee2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:25.214979 7849 consensus_queue.cc:1035] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "590e1ec5c8b645d78fbbcb9b42f5df2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:25.217078 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.216679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:33613)
0927 00:39:25.216886 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.216960 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 00:39:25.219213 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.218099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:58995)
0927 00:39:25.219028 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.219106 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834}
I20260927 00:39:25.221657 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.221230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.1:39811)
0927 00:39:25.221480 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.221546 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260927 00:39:25.224151 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.221935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.3:43981)
0927 00:39:25.223951 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.224034 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1911}
I20260927 00:39:25.224606 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.222165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:45837)
0927 00:39:25.224444 (+ 2279us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.224519 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2179}
I20260927 00:39:25.226389 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43569 (local address 127.0.98.2:39349)
0927 00:39:25.226269 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.226320 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:39:25.231609 7284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:25.231781 7284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:25.236428 7284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d62f7363a6d84930a5bfef65fc465bc4: Sending DeleteTablet for 3 replicas of tablet 38da0334af12471ba2c426fe33399059
I20260927 00:39:25.237180 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet 38da0334af12471ba2c426fe33399059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53952
I20260927 00:39:25.237502 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 38da0334af12471ba2c426fe33399059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55336
I20260927 00:39:25.237586 7864 tablet_replica.cc:330] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: stopping tablet replica
I20260927 00:39:25.237694 7864 raft_consensus.cc:2229] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:25.237828 7865 tablet_replica.cc:330] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b: stopping tablet replica
I20260927 00:39:25.237847 7864 raft_consensus.cc:2258] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:25.238051 7865 raft_consensus.cc:2229] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:25.238049 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 38da0334af12471ba2c426fe33399059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:25 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59518
I20260927 00:39:25.238194 7865 raft_consensus.cc:2258] T 38da0334af12471ba2c426fe33399059 P 590e1ec5c8b645d78fbbcb9b42f5df2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:25.238658 7866 tablet_replica.cc:330] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: stopping tablet replica
I20260927 00:39:25.238824 7866 raft_consensus.cc:2229] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:25.238971 7864 ts_tablet_manager.cc:1905] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:25.239301 7866 raft_consensus.cc:2258] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:25.239413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7247
I20260927 00:39:25.241120 7866 ts_tablet_manager.cc:1905] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:25.243410 7864 ts_tablet_manager.cc:1918] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:25.243505 7864 log.cc:1196] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/38da0334af12471ba2c426fe33399059
I20260927 00:39:25.243736 7864 ts_tablet_manager.cc:1939] T 38da0334af12471ba2c426fe33399059 P 24c56ed0e50043768cb9c0c1d059ee2c: Deleting consensus metadata
I20260927 00:39:25.244122 7866 ts_tablet_manager.cc:1918] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:25.244300 7866 log.cc:1196] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/38da0334af12471ba2c426fe33399059
I20260927 00:39:25.244649 7866 ts_tablet_manager.cc:1939] T 38da0334af12471ba2c426fe33399059 P f69de60d50bb446cbb09afa2638bf5bb: Deleting consensus metadata
I20260927 00:39:25.244613 7272 catalog_manager.cc:4841] TS 24c56ed0e50043768cb9c0c1d059ee2c (127.0.98.3:37465): tablet 38da0334af12471ba2c426fe33399059 (table test-table [id=004ece52da0a423f8b163689a8f3f3a9]) successfully deleted
I20260927 00:39:25.245437 7269 catalog_manager.cc:4841] TS f69de60d50bb446cbb09afa2638bf5bb (127.0.98.2:44685): tablet 38da0334af12471ba2c426fe33399059 (table test-table [id=004ece52da0a423f8b163689a8f3f3a9]) successfully deleted
I20260927 00:39:25.248072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7472
I20260927 00:39:25.248175 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:25.247887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34209 (local address 127.0.0.1:55340)
0927 00:39:25.248022 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:25.248081 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:39:25.249027 7272 catalog_manager.cc:4584] TS 590e1ec5c8b645d78fbbcb9b42f5df2b (127.0.98.1:34209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38da0334af12471ba2c426fe33399059: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34209: connect: Connection refused (error 111)
I20260927 00:39:25.253822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7608
I20260927 00:39:25.258951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 6965
I20260927 00:39:25.264050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7067
2026-09-27T00:39:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7043 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:25 dist-test-slave-xltq krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:25 dist-test-slave-xltq krb5kdc[7869](info): set up 2 sockets
Sep 27 00:39:25 dist-test-slave-xltq krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260927 00:39:26.228659 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:39:26.351788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:26.461833 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:26.462123 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:26.462189 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:26.466361 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:26.466445 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:39:26.466470 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:26.466492 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:26.466513 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:26.466542 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:26.472138 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36521
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:26.473332 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:26.474401 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:26.483474 7882 server_base.cc:1034] running on GCE node
W20260927 00:39:26.483433 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:26.483445 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:26.483451 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:26.484432 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:26.485705 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:26.486868 7882 hybrid_clock.cc:648] HybridClock initialized: now 1790469566486844 us; error 37 us; skew 500 ppm
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:26.489672 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:39:26.490629 7882 webserver.cc:460] Webserver started at http://127.0.98.62:32951/ using document root <none> and password file <none>
I20260927 00:39:26.491178 7882 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:26.491245 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:26.491451 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:26.493642 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "8f53c53ef43d43968b37dd0f452ba246"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "59c8cd45cc93ed2c566b24693fe629c6"
server_key_iv: "ab9c3e47b6e22fbe77d1cd6cf307e9cb"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.494247 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f53c53ef43d43968b37dd0f452ba246"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "59c8cd45cc93ed2c566b24693fe629c6"
server_key_iv: "ab9c3e47b6e22fbe77d1cd6cf307e9cb"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.498229 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:26.500973 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:26.502344 7882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:26.502478 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "8f53c53ef43d43968b37dd0f452ba246"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "59c8cd45cc93ed2c566b24693fe629c6"
server_key_iv: "ab9c3e47b6e22fbe77d1cd6cf307e9cb"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.502602 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:26.519973 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:26.523375 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:26.523571 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:26.531271 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36521
I20260927 00:39:26.531376 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36521 every 8 connection(s)
I20260927 00:39:26.532322 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:26.535473 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:26.538208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7882
I20260927 00:39:26.538336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:26.538635 392 external_mini_cluster.cc:1428] Setting key 73e2e76fe6b9c7067c410e4315cc03ec
I20260927 00:39:26.543263 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Bootstrap starting.
I20260927 00:39:26.546190 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:26.547009 7960 log.cc:824] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:26.549584 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: No bootstrap required, opened a new log
I20260927 00:39:26.553117 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260927 00:39:26.553380 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:26.553462 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f53c53ef43d43968b37dd0f452ba246, State: Initialized, Role: FOLLOWER
I20260927 00:39:26.553927 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260927 00:39:26.554064 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:26.554126 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:26.554219 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:26.555152 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260927 00:39:26.555200 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:26.540705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.98.62:36521)
0927 00:39:26.541211 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:26.541230 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:26.541275 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:39:26.542078 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:26.542114 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:26.542556 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:26.542947 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:26.543924 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.545422 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.546464 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.547018 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.547175 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:26.550204 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:26.550210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:26.550227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:26.550257 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:26.551899 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.552407 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.552413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.552419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.552542 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.552821 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.552824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.552825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.553265 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:26.553471 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:26.554089 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 00:39:26.554412 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260927 00:39:26.555512 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f53c53ef43d43968b37dd0f452ba246; no voters:
I20260927 00:39:26.555815 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:26.555920 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:26.556108 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [term 1 LEADER]: Becoming Leader. State: Replica: 8f53c53ef43d43968b37dd0f452ba246, State: Running, Role: LEADER
I20260927 00:39:26.556491 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260927 00:39:26.557133 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:26.557622 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f53c53ef43d43968b37dd0f452ba246. Latest consensus state: current_term: 1 leader_uuid: "8f53c53ef43d43968b37dd0f452ba246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260927 00:39:26.557744 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:26.558225 7970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:26.558363 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f53c53ef43d43968b37dd0f452ba246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f53c53ef43d43968b37dd0f452ba246" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260927 00:39:26.558449 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:26.561348 7970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:26.566802 7970 catalog_manager.cc:1347] Generated new cluster ID: 348f2ddc68534ef28e7647e8cbb740c4
I20260927 00:39:26.566964 7970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:26.584164 7970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:26.585553 7970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:26.591653 7970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Generated new TSK 0
I20260927 00:39:26.592595 7970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:39:26.624289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:26.741398 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:26.741732 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:26.741797 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:26.746155 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:26.746263 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:26.746320 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:39:26.746376 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:26.752163 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:26.753401 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:26.754527 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:26.765892 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:26.765916 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:26.765916 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:26.766215 7986 server_base.cc:1034] running on GCE node
I20260927 00:39:26.766670 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:26.767407 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:26.768764 7986 hybrid_clock.cc:648] HybridClock initialized: now 1790469566768713 us; error 61 us; skew 500 ppm
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:26.771750 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:39:26.772825 7986 webserver.cc:460] Webserver started at http://127.0.98.1:46225/ using document root <none> and password file <none>
I20260927 00:39:26.773425 7986 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:26.773501 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:26.773715 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:26.776245 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "392ea077573f8af47cce3a8f7aba6d88"
server_key_iv: "33c5797ccdf5e342799d771f616c4dd6"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.776911 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "392ea077573f8af47cce3a8f7aba6d88"
server_key_iv: "33c5797ccdf5e342799d771f616c4dd6"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.781682 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:39:26.784919 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:26.786586 7986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:39:26.786726 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "392ea077573f8af47cce3a8f7aba6d88"
server_key_iv: "33c5797ccdf5e342799d771f616c4dd6"
server_key_version: "encryptionkey@0"
I20260927 00:39:26.786872 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:26.800585 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:26.804086 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:26.804359 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:26.806018 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:26.806074 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:26.806118 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:26.806146 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:26.817025 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46541
I20260927 00:39:26.817050 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46541 every 8 connection(s)
I20260927 00:39:26.818272 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:26.820755 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:26.806903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33087 (local address 127.0.98.62:36521)
0927 00:39:26.807024 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:26.807028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:26.808402 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 00:39:26.809718 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:26.809721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:26.809781 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:26.809856 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:26.811389 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.812014 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.813355 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.813730 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.813870 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:26.816851 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:26.816855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:26.816858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:26.816891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:26.818541 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.819210 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.819214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.819216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.819322 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.819648 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.819650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.819652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.819836 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:26.819909 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:26.820504 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 00:39:26.820628 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:39:26.821494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 7986
I20260927 00:39:26.821621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:26.821684 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:26.807638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.1:33087)
0927 00:39:26.808184 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:26.808284 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:26.809451 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:26.810060 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:26.810070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:26.810484 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:26.811231 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:26.811253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.812141 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.812144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:26.813227 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:26.813238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.813910 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.813914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:26.814086 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:26.814874 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:26.814892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:26.816559 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:26.818699 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:26.818706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:26.818724 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:26.819074 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:26.819416 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:26.819418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:26.819420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:26.819566 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:26.819921 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:26.819926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:26.820180 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 00:39:26.820420 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 00:39:26.820706 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":151,"threads_started":1}
I20260927 00:39:26.822018 392 external_mini_cluster.cc:1428] Setting key 13048a5d7d15a0de56e410a5509047a2
I20260927 00:39:26.823068 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260927 00:39:26.823235 8117 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:26.824007 8117 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260927 00:39:26.825640 7913 ts_manager.cc:194] Registered new tserver with Master: d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541)
I20260927 00:39:26.827206 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33087
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:26.833245 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:26.825587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.1:59711)
0927 00:39:26.825938 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:26.825951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:26.826054 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:26.826284 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:26.826287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:26.826498 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:26.826603 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:26.826609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.827348 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.827352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:26.827967 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:26.827978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.828597 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.828600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:26.828691 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:26.829344 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:26.829359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:26.829813 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:26.831755 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:26.831758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:26.831760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:26.831954 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:26.832282 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:26.832285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:26.832287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:26.832379 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:26.832783 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:26.832786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:26.832874 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:39:26.832958 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:39:26.833083 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":38,"thread_start_us":96,"threads_started":1}
I20260927 00:39:26.833282 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:26.825674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59711 (local address 127.0.98.62:36521)
0927 00:39:26.825789 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:26.825793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:26.826028 (+ 235us) server_negotiation.cc:407] Connection header received
0927 00:39:26.826130 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:26.826133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:26.826177 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:26.826240 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:26.826709 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.827221 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.828073 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:26.828444 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:26.828561 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:26.829979 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:26.829983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:26.829986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:26.830013 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:26.831611 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.832050 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.832053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.832055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.832146 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:26.832450 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:26.832453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:26.832454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:26.832669 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:26.832749 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:26.833006 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 00:39:26.833101 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:39:26.852766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:26.967504 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:26.967895 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:26.967957 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:26.972546 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:26.972626 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:26.972667 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:39:26.972714 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:26.978631 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:26.979769 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:26.980912 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:26.991432 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:26.991520 8126 server_base.cc:1034] running on GCE node
W20260927 00:39:26.991539 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:26.991403 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:26.992168 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:26.992928 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:26.994174 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790469566994147 us; error 39 us; skew 500 ppm
Sep 27 00:39:26 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:26.997316 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:39:26.998358 8126 webserver.cc:460] Webserver started at http://127.0.98.2:38137/ using document root <none> and password file <none>
I20260927 00:39:26.998956 8126 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:26.999028 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:26.999244 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:27.001534 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "026504d0edef4ea081b3385d5f036add"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "04e613abae3c5b9833c77d8e85a779b0"
server_key_iv: "2287af95d90c35ff77eee8b658b8bf72"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.002250 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "026504d0edef4ea081b3385d5f036add"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "04e613abae3c5b9833c77d8e85a779b0"
server_key_iv: "2287af95d90c35ff77eee8b658b8bf72"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.006438 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:27.009248 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:27.010771 8126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:27.010896 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "026504d0edef4ea081b3385d5f036add"
format_stamp: "Formatted at 2026-09-27 00:39:26 on dist-test-slave-xltq"
server_key: "04e613abae3c5b9833c77d8e85a779b0"
server_key_iv: "2287af95d90c35ff77eee8b658b8bf72"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.011010 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:27.020346 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:27.023846 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:27.024034 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:27.025615 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:27.025673 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:27.025727 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:27.025755 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:27.036109 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46801
I20260927 00:39:27.036258 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46801 every 8 connection(s)
I20260927 00:39:27.037395 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:27.039602 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.026539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49399 (local address 127.0.98.62:36521)
0927 00:39:27.026684 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.026689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.027579 (+ 890us) server_negotiation.cc:407] Connection header received
0927 00:39:27.028770 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.028773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.028835 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.028907 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.030347 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.030952 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.032159 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.032436 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.032528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.035375 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.035379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.035382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.035417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.037201 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.037948 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.037952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.037953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.038043 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.038408 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.038411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.038413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.038666 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.038765 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.039284 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.039437 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:39:27.039619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8126
I20260927 00:39:27.039829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:27.040153 392 external_mini_cluster.cc:1428] Setting key 2ecc3981841671b219ed57a4af8d539a
I20260927 00:39:27.040707 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.026940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.2:49399)
0927 00:39:27.027405 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.027457 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.028514 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.029154 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.029165 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.029572 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:27.030134 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.030150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.031084 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.031087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.032027 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.032040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.032611 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.032616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.032798 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.033440 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:27.033457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:27.035153 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:27.037374 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.037380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.037392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.037811 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.038129 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.038133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.038135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.038312 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.038767 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:27.038774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:27.039039 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.039224 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.039571 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260927 00:39:27.043347 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260927 00:39:27.043529 8257 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:27.044106 8257 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260927 00:39:27.045404 7914 ts_manager.cc:194] Registered new tserver with Master: 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801)
I20260927 00:39:27.046178 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49399
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:27.052742 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.044807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42569 (local address 127.0.98.62:36521)
0927 00:39:27.044940 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.044945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.045200 (+ 255us) server_negotiation.cc:407] Connection header received
0927 00:39:27.045316 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.045320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.045364 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.045430 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.045899 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.046520 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.047484 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.047706 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.047801 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.049265 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.049270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.049273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.049304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.050978 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.051497 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.051499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.051500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.051582 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.051838 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.051840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.051841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.052071 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.052153 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.052497 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.052605 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:39:27.052758 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.044728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.2:42569)
0927 00:39:27.045108 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.045124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.045231 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.045459 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.045462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.045663 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:27.045781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.045787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.046658 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.046662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.047353 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.047364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.047821 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.047825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.047916 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.048603 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:27.048618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:27.049076 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:27.051141 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.051144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.051146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.051385 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.051664 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.051667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.051668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.051752 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.052227 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:27.052230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:27.052333 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.052432 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.052565 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":4,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:39:27.073071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:27.189924 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:27.190209 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:27.190256 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:27.194576 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:27.194661 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:27.194703 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:39:27.194737 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:27.200309 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:27.201381 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:27.202386 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:27.212742 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:27.212690 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:27.212710 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:27.213086 8266 server_base.cc:1034] running on GCE node
I20260927 00:39:27.213589 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:27.214452 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:27.215646 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790469567215617 us; error 45 us; skew 500 ppm
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469567, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:27.218593 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:39:27.219643 8266 webserver.cc:460] Webserver started at http://127.0.98.3:43949/ using document root <none> and password file <none>
I20260927 00:39:27.220244 8266 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:27.220316 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:27.220541 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:27.222730 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "d559eb6879274181ac611a0fdb10397c"
format_stamp: "Formatted at 2026-09-27 00:39:27 on dist-test-slave-xltq"
server_key: "17ee0112c3ec32f1c6e3035b7d8cd081"
server_key_iv: "aca6e6f228135f0f279dc1ec7da58f8c"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.223364 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d559eb6879274181ac611a0fdb10397c"
format_stamp: "Formatted at 2026-09-27 00:39:27 on dist-test-slave-xltq"
server_key: "17ee0112c3ec32f1c6e3035b7d8cd081"
server_key_iv: "aca6e6f228135f0f279dc1ec7da58f8c"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.227501 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:27.230360 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:27.231706 8266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:27.231823 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "d559eb6879274181ac611a0fdb10397c"
format_stamp: "Formatted at 2026-09-27 00:39:27 on dist-test-slave-xltq"
server_key: "17ee0112c3ec32f1c6e3035b7d8cd081"
server_key_iv: "aca6e6f228135f0f279dc1ec7da58f8c"
server_key_version: "encryptionkey@0"
I20260927 00:39:27.231933 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:27.249248 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:27.252750 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:27.252964 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:27.254649 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:27.254726 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:27.254781 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:27.254810 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469567, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:27.265727 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45593
I20260927 00:39:27.265780 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45593 every 8 connection(s)
I20260927 00:39:27.266824 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:27.269531 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.255583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38649 (local address 127.0.98.62:36521)
0927 00:39:27.255813 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.255817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.256823 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 00:39:27.258266 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.258271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.258329 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.258427 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.259853 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.260509 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.261767 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.261981 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.262073 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.265689 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.265694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.265697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.265733 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.267385 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.267999 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.268003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.268004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.268094 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.268437 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.268439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.268440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.268638 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.268735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.269298 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.269409 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:39:27.269917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8266
I20260927 00:39:27.270022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:27.270285 392 external_mini_cluster.cc:1428] Setting key 3dc42b38e9c618dbecc9297157a6faab
I20260927 00:39:27.270471 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.256083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.3:38649)
0927 00:39:27.256642 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.256707 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.257989 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.258605 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.258612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.259086 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:27.259705 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.259722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.260631 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.260634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.261639 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.261658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.262139 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.262143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.262324 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.263529 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:27.263549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:27.265345 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:27.267552 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.267557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.267575 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.267897 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.268176 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.268178 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.268179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.268347 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.268740 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:27.268805 (+ 65us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:27.269055 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.269214 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.269490 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":125,"threads_started":1}
I20260927 00:39:27.271703 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260927 00:39:27.271843 8397 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:27.272370 8397 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260927 00:39:27.273344 7914 ts_manager.cc:194] Registered new tserver with Master: d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:27.273895 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38649
I20260927 00:39:27.280930 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.274175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60913 (local address 127.0.98.62:36521)
0927 00:39:27.274286 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.274291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.274542 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:39:27.274644 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.274647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.274691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.274756 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.275217 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.275720 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.276637 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.276861 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.276950 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.278047 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.278050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.278053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.278078 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.279351 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.279744 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.279747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.279748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.279823 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.280066 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.280070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.280071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.280297 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.280385 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.280651 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.280770 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:27.281054 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.274126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.3:60913)
0927 00:39:27.274454 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.274468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.274569 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.274784 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.274787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.274994 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:27.275101 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.275107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.275839 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.275843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.276523 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.276535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.276964 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.276968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.277060 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.277550 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:27.277563 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:27.277908 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:27.279456 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.279459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.279461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.279649 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.279904 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.279907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.279909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.279988 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.280382 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:27.280385 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:27.280499 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.280586 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.280866 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":146,"threads_started":1}
I20260927 00:39:27.284636 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:39:27.308589 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:27.301769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.0.98.62:36521)
0927 00:39:27.301899 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.301903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.301916 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:27.301959 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.301962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.302003 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.302143 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.302878 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.303411 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.304453 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.304678 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.304784 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.308460 (+ 3676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39096: BlockingRecv error: recv got EOF from 127.0.0.1:39096 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:27.318729 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.310093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.0.98.62:36521)
0927 00:39:27.310235 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.310239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.310306 (+ 67us) server_negotiation.cc:407] Connection header received
0927 00:39:27.310433 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.310436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.310478 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.310542 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.311264 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.311831 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.312750 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.312959 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.313038 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.315631 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.315640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.315643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.315675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.317083 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.317507 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.317510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.317511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.317597 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.317904 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.317906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.317907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.318095 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.318250 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.318491 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.318592 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:39:27.321933 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:27.325400 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:27.340487 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.335540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.98.1:46541)
0927 00:39:27.335939 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.335946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.335969 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:39:27.336046 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.336052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.336331 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.336834 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.336984 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.337781 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.339167 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.339794 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.339899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.340049 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.340240 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.340342 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":78,"threads_started":1}
I20260927 00:39:27.340507 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.334721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.0.1:48750)
0927 00:39:27.335124 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.335233 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.335365 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.336499 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.336506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.336528 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.336673 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.336680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.337886 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.337890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.339055 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.339064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.339912 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.339915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.340020 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.340064 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.340179 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.340357 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260927 00:39:27.340668 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.334838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.0.1:60458)
0927 00:39:27.335727 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.335740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.335828 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.336779 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.336782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.336799 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.336884 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.336890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.338220 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.338223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.339393 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.339402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.340357 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.340360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.340441 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.340461 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.340505 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.340564 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260927 00:39:27.341080 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.335450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.0.1:44690)
0927 00:39:27.335995 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.336006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.336071 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.336643 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.336645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.336659 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.336725 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.336730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.338141 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.338144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.339533 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.339541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.340729 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.340733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.340819 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.340842 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.340911 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.340980 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":250,"thread_start_us":83,"threads_started":1}
I20260927 00:39:27.341296 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.334981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.2:46801)
0927 00:39:27.336220 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.336231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.336248 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:39:27.336317 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.336321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.336488 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.336639 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.337007 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.338118 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.339526 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.340244 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.340849 (+ 605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.341008 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.341054 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.341155 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":86,"threads_started":1}
I20260927 00:39:27.341475 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.335693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.0.98.3:45593)
0927 00:39:27.336146 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.336158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.336180 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:39:27.336273 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.336278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.336488 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.336639 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.336832 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.338015 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.339649 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.340422 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.340518 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.340668 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.341246 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.341345 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":95,"threads_started":1}
I20260927 00:39:27.343950 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 61554d5836824e2d92168a646e1aeb5c (DEFAULT_TABLE table=test-table [id=2b57e59bd09a40f499d6751889f4b0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.344055 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 61554d5836824e2d92168a646e1aeb5c (DEFAULT_TABLE table=test-table [id=2b57e59bd09a40f499d6751889f4b0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.344448 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 61554d5836824e2d92168a646e1aeb5c (DEFAULT_TABLE table=test-table [id=2b57e59bd09a40f499d6751889f4b0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.344960 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61554d5836824e2d92168a646e1aeb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.345083 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61554d5836824e2d92168a646e1aeb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.345341 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61554d5836824e2d92168a646e1aeb5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.351328 8425 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Bootstrap starting.
I20260927 00:39:27.352422 8427 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Bootstrap starting.
I20260927 00:39:27.353160 8426 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Bootstrap starting.
I20260927 00:39:27.353837 8425 tablet_bootstrap.cc:654] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.354746 8425 log.cc:824] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:27.355370 8426 tablet_bootstrap.cc:654] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.356273 8426 log.cc:824] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:27.356997 8425 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: No bootstrap required, opened a new log
I20260927 00:39:27.357200 8425 ts_tablet_manager.cc:1397] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:39:27.357316 8427 tablet_bootstrap.cc:654] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.358290 8427 log.cc:824] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:27.358417 8426 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: No bootstrap required, opened a new log
I20260927 00:39:27.358600 8426 ts_tablet_manager.cc:1397] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:27.360497 8427 tablet_bootstrap.cc:492] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: No bootstrap required, opened a new log
I20260927 00:39:27.360702 8427 ts_tablet_manager.cc:1397] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:39:27.361517 8425 raft_consensus.cc:348] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.361822 8425 raft_consensus.cc:374] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.361689 8426 raft_consensus.cc:348] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.361878 8425 raft_consensus.cc:729] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.361902 8426 raft_consensus.cc:374] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.361945 8426 raft_consensus.cc:729] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.362341 8425 consensus_queue.cc:260] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.362550 8426 consensus_queue.cc:260] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.363200 8117 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260927 00:39:27.363397 8425 ts_tablet_manager.cc:1428] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:39:27.363581 8426 ts_tablet_manager.cc:1428] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:39:27.363930 8397 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260927 00:39:27.365540 8427 raft_consensus.cc:348] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.365808 8427 raft_consensus.cc:374] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.365869 8427 raft_consensus.cc:729] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.366339 8427 consensus_queue.cc:260] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.367026 8257 heartbeater.cc:503] Master 127.0.98.62:36521 was elected leader, sending a full tablet report...
I20260927 00:39:27.367372 8427 ts_tablet_manager.cc:1428] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:39:27.405037 8431 raft_consensus.cc:482] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:27.405283 8431 raft_consensus.cc:504] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.406500 8431 leader_election.cc:288] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:27.411255 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.406868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60853 (local address 127.0.98.2:46801)
0927 00:39:27.406988 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.406992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.407238 (+ 246us) server_negotiation.cc:407] Connection header received
0927 00:39:27.407377 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.407379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.407426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.407486 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.407779 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.408380 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.409974 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.410561 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.410670 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.410733 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.411059 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.411126 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:39:27.411356 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.406813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.98.1:56757)
0927 00:39:27.407248 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.407259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.407390 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.407633 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.407635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.407649 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.407782 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.407788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.408671 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.408674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.409864 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.409870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.410709 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.410713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.410919 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.410956 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.411040 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.411173 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":166,"thread_start_us":120,"threads_started":1}
I20260927 00:39:27.411585 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.406756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.98.1:60853)
0927 00:39:27.407142 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.407156 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.407277 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.407518 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.407520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.407534 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.407679 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.407686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.408494 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.408497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.409858 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.409870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.410722 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.410725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.410905 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.410960 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.411038 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.411171 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":100,"thread_start_us":118,"threads_started":1}
I20260927 00:39:27.411839 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.406912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56757 (local address 127.0.98.3:45593)
0927 00:39:27.407035 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.407119 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.407369 (+ 250us) server_negotiation.cc:407] Connection header received
0927 00:39:27.407488 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.407491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.407542 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.407609 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.407897 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.408589 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.409974 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.410561 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.410670 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.410733 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.411471 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.411538 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:39:27.412228 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61554d5836824e2d92168a646e1aeb5c" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add" is_pre_election: true
I20260927 00:39:27.412341 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61554d5836824e2d92168a646e1aeb5c" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c" is_pre_election: true
I20260927 00:39:27.412530 8211 raft_consensus.cc:2454] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 0.
I20260927 00:39:27.412600 8351 raft_consensus.cc:2454] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 0.
I20260927 00:39:27.413022 8005 leader_election.cc:302] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:27.413300 8431 raft_consensus.cc:2781] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:27.413386 8431 raft_consensus.cc:482] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:27.413435 8431 raft_consensus.cc:3037] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.414455 8431 raft_consensus.cc:504] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.414829 8431 leader_election.cc:288] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 election: Requested vote from peers 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:27.415208 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61554d5836824e2d92168a646e1aeb5c" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add"
I20260927 00:39:27.415202 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61554d5836824e2d92168a646e1aeb5c" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c"
I20260927 00:39:27.415340 8211 raft_consensus.cc:3037] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.415372 8351 raft_consensus.cc:3037] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.416674 8351 raft_consensus.cc:2454] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 1.
I20260927 00:39:27.416674 8211 raft_consensus.cc:2454] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 1.
I20260927 00:39:27.417021 8002 leader_election.cc:302] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d559eb6879274181ac611a0fdb10397c, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:27.417205 8431 raft_consensus.cc:2781] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:27.417416 8431 raft_consensus.cc:686] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 LEADER]: Becoming Leader. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Running, Role: LEADER
I20260927 00:39:27.417810 8431 consensus_queue.cc:237] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.421018 7914 catalog_manager.cc:5495] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:27.441062 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.437417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.98.1:46541)
0927 00:39:27.437572 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.437576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.437590 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:27.437734 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.437738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.437802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.437898 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:27.438106 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.438653 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.439497 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.439786 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.439870 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.440095 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:27.440638 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:27.440805 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.440862 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.440919 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:39:27.447043 8211 raft_consensus.cc:1261] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Refusing update from remote peer d7c76377be1749b2bafdf8b4b468e2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.447307 8351 raft_consensus.cc:1261] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Refusing update from remote peer d7c76377be1749b2bafdf8b4b468e2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.447773 8431 consensus_queue.cc:1035] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.448019 8436 consensus_queue.cc:1035] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.463882 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:27.464088 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:27.466813 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Sending DeleteTablet for 3 replicas of tablet 61554d5836824e2d92168a646e1aeb5c
I20260927 00:39:27.467643 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 61554d5836824e2d92168a646e1aeb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60458
I20260927 00:39:27.467695 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 61554d5836824e2d92168a646e1aeb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48750
I20260927 00:39:27.467694 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 61554d5836824e2d92168a646e1aeb5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44690
I20260927 00:39:27.468312 8450 tablet_replica.cc:330] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: stopping tablet replica
I20260927 00:39:27.468353 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:39:27.468653 8451 tablet_replica.cc:330] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: stopping tablet replica
I20260927 00:39:27.468667 8450 raft_consensus.cc:2229] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 00:39:27.468779 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:27.469020 8450 raft_consensus.cc:2258] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:27.469063 8451 raft_consensus.cc:2229] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:27.469498 8451 raft_consensus.cc:2258] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:27.469573 8449 tablet_replica.cc:330] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: stopping tablet replica
I20260927 00:39:27.470475 8449 raft_consensus.cc:2229] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:27.470908 8449 raft_consensus.cc:2258] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:27.470965 8450 ts_tablet_manager.cc:1905] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:27.471282 8451 ts_tablet_manager.cc:1905] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:27.473623 8449 ts_tablet_manager.cc:1905] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:27.473845 8450 ts_tablet_manager.cc:1918] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:27.473953 8450 log.cc:1196] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/61554d5836824e2d92168a646e1aeb5c
I20260927 00:39:27.474172 8451 ts_tablet_manager.cc:1918] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:27.474284 8450 ts_tablet_manager.cc:1939] T 61554d5836824e2d92168a646e1aeb5c P d559eb6879274181ac611a0fdb10397c: Deleting consensus metadata
I20260927 00:39:27.474408 8451 log.cc:1196] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/61554d5836824e2d92168a646e1aeb5c
I20260927 00:39:27.474764 8451 ts_tablet_manager.cc:1939] T 61554d5836824e2d92168a646e1aeb5c P 026504d0edef4ea081b3385d5f036add: Deleting consensus metadata
I20260927 00:39:27.474884 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 (DEFAULT_TABLE table=test-table [id=ea1fe8e6090844ea814ab974f239e470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.475219 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057b70cf06a94c9083cea17c8408c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.475581 7898 catalog_manager.cc:4841] TS d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593): tablet 61554d5836824e2d92168a646e1aeb5c (table test-table [id=2b57e59bd09a40f499d6751889f4b0e2]) successfully deleted
I20260927 00:39:27.476115 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 (DEFAULT_TABLE table=test-table [id=ea1fe8e6090844ea814ab974f239e470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.476368 8449 ts_tablet_manager.cc:1918] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:27.476428 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057b70cf06a94c9083cea17c8408c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.476413 7901 catalog_manager.cc:4841] TS 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801): tablet 61554d5836824e2d92168a646e1aeb5c (table test-table [id=2b57e59bd09a40f499d6751889f4b0e2]) successfully deleted
I20260927 00:39:27.476468 8449 log.cc:1196] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/61554d5836824e2d92168a646e1aeb5c
I20260927 00:39:27.476759 8449 ts_tablet_manager.cc:1939] T 61554d5836824e2d92168a646e1aeb5c P d7c76377be1749b2bafdf8b4b468e2d1: Deleting consensus metadata
I20260927 00:39:27.478027 8425 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Bootstrap starting.
I20260927 00:39:27.478935 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 (DEFAULT_TABLE table=test-table [id=ea1fe8e6090844ea814ab974f239e470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:27.479231 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057b70cf06a94c9083cea17c8408c6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.480315 7898 catalog_manager.cc:4841] TS d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541): tablet 61554d5836824e2d92168a646e1aeb5c (table test-table [id=2b57e59bd09a40f499d6751889f4b0e2]) successfully deleted
I20260927 00:39:27.481626 8426 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Bootstrap starting.
I20260927 00:39:27.481720 8427 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Bootstrap starting.
I20260927 00:39:27.482735 8427 tablet_bootstrap.cc:654] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.483338 8426 tablet_bootstrap.cc:654] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.483393 8425 tablet_bootstrap.cc:654] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.484854 8427 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: No bootstrap required, opened a new log
I20260927 00:39:27.484956 8427 ts_tablet_manager.cc:1397] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:27.485450 8427 raft_consensus.cc:348] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.485579 8427 raft_consensus.cc:374] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.485615 8427 raft_consensus.cc:729] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.485740 8427 consensus_queue.cc:260] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.486004 8427 ts_tablet_manager.cc:1428] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:27.487433 8426 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: No bootstrap required, opened a new log
I20260927 00:39:27.487531 8426 ts_tablet_manager.cc:1397] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 00:39:27.488030 8425 tablet_bootstrap.cc:492] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: No bootstrap required, opened a new log
I20260927 00:39:27.488021 8426 raft_consensus.cc:348] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.488118 8425 ts_tablet_manager.cc:1397] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 00:39:27.488143 8426 raft_consensus.cc:374] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.488193 8426 raft_consensus.cc:729] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.488339 8426 consensus_queue.cc:260] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.488607 8426 ts_tablet_manager.cc:1428] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:27.488605 8425 raft_consensus.cc:348] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.488730 8425 raft_consensus.cc:374] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.488761 8425 raft_consensus.cc:729] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.489027 8425 consensus_queue.cc:260] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.489279 8425 ts_tablet_manager.cc:1428] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260927 00:39:27.517912 8398 tablet.cc:2369] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:39:27.539198 8258 tablet.cc:2369] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:39:27.569852 8118 tablet.cc:2369] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:39:27.693636 8445 raft_consensus.cc:482] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:27.693785 8445 raft_consensus.cc:504] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.694206 8445 leader_election.cc:288] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:27.694726 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057b70cf06a94c9083cea17c8408c6c3" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c" is_pre_election: true
I20260927 00:39:27.694778 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057b70cf06a94c9083cea17c8408c6c3" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add" is_pre_election: true
I20260927 00:39:27.694901 8211 raft_consensus.cc:2454] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 0.
I20260927 00:39:27.694900 8351 raft_consensus.cc:2454] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 0.
I20260927 00:39:27.695257 8002 leader_election.cc:302] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d559eb6879274181ac611a0fdb10397c, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:27.695446 8445 raft_consensus.cc:2781] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:27.695513 8445 raft_consensus.cc:482] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:27.695538 8445 raft_consensus.cc:3037] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.696568 8445 raft_consensus.cc:504] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.696930 8445 leader_election.cc:288] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 election: Requested vote from peers 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:27.697342 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057b70cf06a94c9083cea17c8408c6c3" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add"
I20260927 00:39:27.697336 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057b70cf06a94c9083cea17c8408c6c3" candidate_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c"
I20260927 00:39:27.697471 8351 raft_consensus.cc:3037] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.697472 8211 raft_consensus.cc:3037] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.698614 8351 raft_consensus.cc:2454] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 1.
I20260927 00:39:27.698613 8211 raft_consensus.cc:2454] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c76377be1749b2bafdf8b4b468e2d1 in term 1.
I20260927 00:39:27.698968 8005 leader_election.cc:302] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d559eb6879274181ac611a0fdb10397c, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:27.699155 8445 raft_consensus.cc:2781] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:27.699247 8445 raft_consensus.cc:686] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 LEADER]: Becoming Leader. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Running, Role: LEADER
I20260927 00:39:27.699431 8445 consensus_queue.cc:237] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.701117 7913 catalog_manager.cc:5495] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } health_report { overall_health: UNKNOWN } } }
Sep 27 00:39:27 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469566, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:27.766989 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.758383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.62:36521)
0927 00:39:27.758527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.758533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.758751 (+ 218us) server_negotiation.cc:407] Connection header received
0927 00:39:27.758879 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.758882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.758921 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.758988 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:27.759693 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.760257 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.761017 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.761234 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.761320 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.763827 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:27.763830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:27.763832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:27.763855 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:27.765331 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.765814 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.765818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.765819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.765892 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:27.766123 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:27.766126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:27.766128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:27.766318 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:27.766413 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.766700 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.766809 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:39:27.767004 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.758351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.0.1:39114)
0927 00:39:27.758663 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.758675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.758791 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.759021 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.759023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.759457 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:27.759553 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.759560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.760359 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.760363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.760905 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.760914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.761335 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.761338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.761408 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.762026 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:27.762041 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:27.763669 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:27.765449 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.765461 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.765468 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.765714 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.765970 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:27.765973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:27.765975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:27.766049 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:27.766425 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:27.766433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:27.766543 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.766641 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.766875 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260927 00:39:27.772181 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:39:27.773126 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:27.778937 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6dd6459be249dba93a9273dd3b7e5f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2760e405fea84c8fb9954f893bce3fb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:39:27.779003 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6dd6459be249dba93a9273dd3b7e5f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2760e405fea84c8fb9954f893bce3fb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:39:27.779105 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6dd6459be249dba93a9273dd3b7e5f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2760e405fea84c8fb9954f893bce3fb9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:39:27.779390 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6dd6459be249dba93a9273dd3b7e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.779440 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6dd6459be249dba93a9273dd3b7e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.779527 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6dd6459be249dba93a9273dd3b7e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:27.782483 8425 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: Bootstrap starting.
I20260927 00:39:27.782517 8426 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: Bootstrap starting.
I20260927 00:39:27.783442 8425 tablet_bootstrap.cc:654] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.783468 8426 tablet_bootstrap.cc:654] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.784116 8427 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: Bootstrap starting.
I20260927 00:39:27.784655 8425 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: No bootstrap required, opened a new log
I20260927 00:39:27.784746 8425 ts_tablet_manager.cc:1397] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:39:27.784938 8426 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: No bootstrap required, opened a new log
I20260927 00:39:27.785022 8426 ts_tablet_manager.cc:1397] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:39:27.785260 8427 tablet_bootstrap.cc:654] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:27.785197 8425 raft_consensus.cc:348] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.785327 8425 raft_consensus.cc:374] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.785372 8425 raft_consensus.cc:729] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.785508 8425 consensus_queue.cc:260] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.785493 8426 raft_consensus.cc:348] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.785606 8426 raft_consensus.cc:374] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.785645 8426 raft_consensus.cc:729] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.785777 8425 ts_tablet_manager.cc:1428] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:27.785760 8426 consensus_queue.cc:260] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.785735 8445 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.785887 8445 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.786038 8426 ts_tablet_manager.cc:1428] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:39:27.786228 8432 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.786368 8432 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.787732 8427 tablet_bootstrap.cc:492] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: No bootstrap required, opened a new log
I20260927 00:39:27.787838 8427 ts_tablet_manager.cc:1397] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:39:27.788344 8427 raft_consensus.cc:348] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.788444 8427 raft_consensus.cc:374] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:27.788465 8427 raft_consensus.cc:729] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Initialized, Role: FOLLOWER
I20260927 00:39:27.788566 8427 consensus_queue.cc:260] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.788803 8427 ts_tablet_manager.cc:1428] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:27.788913 8433 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.789063 8433 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.909523 8432 raft_consensus.cc:482] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:27.909660 8432 raft_consensus.cc:504] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.910835 8432 leader_election.cc:288] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801)
I20260927 00:39:27.914659 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.911211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48935 (local address 127.0.98.2:46801)
0927 00:39:27.911335 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.911339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.911661 (+ 322us) server_negotiation.cc:407] Connection header received
0927 00:39:27.911770 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.911774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.911822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.911890 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.912219 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.912735 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.913756 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.914141 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.914231 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.914297 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.914462 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.914532 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:27.914697 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.911103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.98.3:48935)
0927 00:39:27.911552 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.911566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.911677 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.911978 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.911981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.911995 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.912086 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.912090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.912838 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.912841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.913639 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.913648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.914254 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.914258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.914359 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.914379 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.914441 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.914514 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":177,"thread_start_us":84,"threads_started":1}
I20260927 00:39:27.914721 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.911226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59665 (local address 127.0.98.1:46541)
0927 00:39:27.911347 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.911351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.911531 (+ 180us) server_negotiation.cc:407] Connection header received
0927 00:39:27.911677 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.911681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.911727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.911797 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:27.912037 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.912619 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.913748 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.914201 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.914329 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.914396 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.914534 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.914609 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:39:27.914791 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.911099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.98.3:59665)
0927 00:39:27.911437 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.911452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.911567 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.911814 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.911817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.911832 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:27.911938 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.911944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.912727 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.912730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.913622 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.913632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.914313 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.914317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.914410 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.914433 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.914498 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.914572 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":68,"threads_started":1}
I20260927 00:39:27.915119 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6dd6459be249dba93a9273dd3b7e5f" candidate_uuid: "d559eb6879274181ac611a0fdb10397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add" is_pre_election: true
I20260927 00:39:27.915293 8211 raft_consensus.cc:2454] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d559eb6879274181ac611a0fdb10397c in term 0.
I20260927 00:39:27.915427 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6dd6459be249dba93a9273dd3b7e5f" candidate_uuid: "d559eb6879274181ac611a0fdb10397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" is_pre_election: true
I20260927 00:39:27.915623 8071 raft_consensus.cc:2454] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d559eb6879274181ac611a0fdb10397c in term 0.
I20260927 00:39:27.915639 8285 leader_election.cc:302] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d559eb6879274181ac611a0fdb10397c; no voters:
I20260927 00:39:27.915884 8432 raft_consensus.cc:2781] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:27.915942 8432 raft_consensus.cc:482] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:27.915979 8432 raft_consensus.cc:3037] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.916991 8432 raft_consensus.cc:504] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.917372 8432 leader_election.cc:288] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [CANDIDATE]: Term 1 election: Requested vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801)
I20260927 00:39:27.917725 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6dd6459be249dba93a9273dd3b7e5f" candidate_uuid: "d559eb6879274181ac611a0fdb10397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
I20260927 00:39:27.917716 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6dd6459be249dba93a9273dd3b7e5f" candidate_uuid: "d559eb6879274181ac611a0fdb10397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026504d0edef4ea081b3385d5f036add"
I20260927 00:39:27.917860 8071 raft_consensus.cc:3037] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.917923 8211 raft_consensus.cc:3037] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:27.919028 8071 raft_consensus.cc:2454] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d559eb6879274181ac611a0fdb10397c in term 1.
I20260927 00:39:27.919028 8211 raft_consensus.cc:2454] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d559eb6879274181ac611a0fdb10397c in term 1.
I20260927 00:39:27.919378 8282 leader_election.cc:302] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d559eb6879274181ac611a0fdb10397c, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:27.919574 8432 raft_consensus.cc:2781] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:27.919775 8432 raft_consensus.cc:686] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [term 1 LEADER]: Becoming Leader. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Running, Role: LEADER
I20260927 00:39:27.920074 8432 consensus_queue.cc:237] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:27.921327 8467 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: TxnStatusTablet state changed. Reason: New leader d559eb6879274181ac611a0fdb10397c. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.921446 8467 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:39:27.921962 8469 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:39:27.922101 8469 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:39:27.922715 7913 catalog_manager.cc:5495] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c reported cstate change: term changed from 0 to 1, leader changed from <none> to d559eb6879274181ac611a0fdb10397c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } health_report { overall_health: HEALTHY } } }
I20260927 00:39:27.946650 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.942848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.0.98.3:45593)
0927 00:39:27.943153 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.943160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.943173 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:27.943236 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.943238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.943289 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.943441 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:27.943686 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.944269 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.945117 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.945355 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.945441 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.945747 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:27.946203 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:27.946308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.946417 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.946475 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260927 00:39:27.946630 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.942750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.0.1:44694)
0927 00:39:27.942903 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.942918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.943034 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.943461 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.943465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.943475 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:27.943575 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.943581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.944366 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.944370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.945008 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.945017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.945456 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.945459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.945547 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.945596 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:27.946297 (+ 701us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:27.946313 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.946407 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.946506 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 00:39:27.950137 8211 raft_consensus.cc:1261] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Refusing update from remote peer d559eb6879274181ac611a0fdb10397c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.950289 8071 raft_consensus.cc:1261] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Refusing update from remote peer d559eb6879274181ac611a0fdb10397c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.950649 8467 consensus_queue.cc:1035] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [LEADER]: Connected to new peer: Peer: permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.950829 8432 consensus_queue.cc:1035] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.952759 8445 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: TxnStatusTablet state changed. Reason: New leader d559eb6879274181ac611a0fdb10397c. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.952804 8433 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: TxnStatusTablet state changed. Reason: New leader d559eb6879274181ac611a0fdb10397c. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.952904 8445 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.952917 8433 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.954391 8467 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.954526 8467 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:39:27.956075 8432 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.956224 8432 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d559eb6879274181ac611a0fdb10397c: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:39:27.957289 8445 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.957408 8445 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P d7c76377be1749b2bafdf8b4b468e2d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.958742 8433 tablet_replica.cc:439] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d559eb6879274181ac611a0fdb10397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } } }
I20260927 00:39:27.958896 8433 tablet_replica.cc:442] T 2a6dd6459be249dba93a9273dd3b7e5f P 026504d0edef4ea081b3385d5f036add: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:39:27.958791 8441 mvcc.cc:205] Tried to move back new op lower bound from 7333763350318211072 to 7333763350203310080. Current Snapshot: MvccSnapshot[applied={T|T < 7333763350318211072}]
I20260927 00:39:27.959939 8440 mvcc.cc:205] Tried to move back new op lower bound from 7333763350318211072 to 7333763350203310080. Current Snapshot: MvccSnapshot[applied={T|T < 7333763350318211072 or (T in {7333763350318211072})}]
I20260927 00:39:27.973426 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.969019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44939 (local address 127.0.98.3:45593)
0927 00:39:27.969147 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:27.969152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:27.969634 (+ 482us) server_negotiation.cc:407] Connection header received
0927 00:39:27.969815 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:27.969818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:27.969897 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:27.969966 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:27.970356 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.971160 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.972090 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.972463 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.972604 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.973007 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:27.973089 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:27.973157 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:27.973244 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:39:27.973297 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:39:27.973497 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:27.968674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.98.1:44939)
0927 00:39:27.969528 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:27.969544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:27.969664 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:27.970079 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:27.970083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:27.970091 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:27.970226 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.970232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.971274 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.971277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.971968 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:27.971985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:27.972699 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:27.972703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:27.972800 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:27.972859 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:27.973168 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:27.973174 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:27.973233 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 00:39:27.973327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":108,"threads_started":1}
I20260927 00:39:27.979954 8211 raft_consensus.cc:1261] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Refusing update from remote peer d7c76377be1749b2bafdf8b4b468e2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.979954 8351 raft_consensus.cc:1261] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Refusing update from remote peer d7c76377be1749b2bafdf8b4b468e2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:27.980403 8431 consensus_queue.cc:1035] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.980567 8445 consensus_queue.cc:1035] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:27.985863 8439 mvcc.cc:205] Tried to move back new op lower bound from 7333763350442795008 to 7333763349297991680. Current Snapshot: MvccSnapshot[applied={T|T < 7333763350442795008}]
I20260927 00:39:28.008500 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.000817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36491 (local address 127.0.98.1:46541)
0927 00:39:28.000970 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.000974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.004558 (+ 3584us) server_negotiation.cc:407] Connection header received
0927 00:39:28.004757 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.004760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.004825 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.004906 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:28.005293 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.006103 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.007158 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.007513 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.007637 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.007983 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.008074 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.008138 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.008330 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.008393 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:39:28.008740 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.000752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.98.3:36491)
0927 00:39:28.004471 (+ 3719us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.004488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:28.004599 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:28.005022 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:28.005026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:28.005034 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:28.005157 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.005164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.006230 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.006234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.007050 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.007061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.007652 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.007654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.007791 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.007848 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.008178 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.008185 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:39:28.008286 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 00:39:28.008373 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3570,"thread_start_us":93,"threads_started":1}
I20260927 00:39:28.030333 7912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.030501 7912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.032761 7912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Sending DeleteTablet for 3 replicas of tablet 057b70cf06a94c9083cea17c8408c6c3
I20260927 00:39:28.033416 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44690
I20260927 00:39:28.033406 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60458
I20260927 00:39:28.034011 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 057b70cf06a94c9083cea17c8408c6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48750
I20260927 00:39:28.034369 8486 tablet_replica.cc:330] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: stopping tablet replica
I20260927 00:39:28.034528 8486 raft_consensus.cc:2229] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.034574 8488 tablet_replica.cc:330] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: stopping tablet replica
I20260927 00:39:28.034708 8486 raft_consensus.cc:2258] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.034890 8488 raft_consensus.cc:2229] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:28.034744 8487 tablet_replica.cc:330] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: stopping tablet replica
I20260927 00:39:28.035151 8487 raft_consensus.cc:2229] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.035494 8486 ts_tablet_manager.cc:1905] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.035399 8487 raft_consensus.cc:2258] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.035653 8488 raft_consensus.cc:2258] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.037099 8488 ts_tablet_manager.cc:1905] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.040046 8488 ts_tablet_manager.cc:1918] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:28.040127 8488 log.cc:1196] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/057b70cf06a94c9083cea17c8408c6c3
I20260927 00:39:28.040382 8488 ts_tablet_manager.cc:1939] T 057b70cf06a94c9083cea17c8408c6c3 P d7c76377be1749b2bafdf8b4b468e2d1: Deleting consensus metadata
I20260927 00:39:28.040688 8487 ts_tablet_manager.cc:1905] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.043553 8487 ts_tablet_manager.cc:1918] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:28.043640 8487 log.cc:1196] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/057b70cf06a94c9083cea17c8408c6c3
I20260927 00:39:28.043936 8487 ts_tablet_manager.cc:1939] T 057b70cf06a94c9083cea17c8408c6c3 P d559eb6879274181ac611a0fdb10397c: Deleting consensus metadata
I20260927 00:39:28.044032 8486 ts_tablet_manager.cc:1918] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:28.044162 8486 log.cc:1196] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/057b70cf06a94c9083cea17c8408c6c3
I20260927 00:39:28.044543 8486 ts_tablet_manager.cc:1939] T 057b70cf06a94c9083cea17c8408c6c3 P 026504d0edef4ea081b3385d5f036add: Deleting consensus metadata
I20260927 00:39:28.045466 7901 catalog_manager.cc:4841] TS 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801): tablet 057b70cf06a94c9083cea17c8408c6c3 (table test-table [id=ea1fe8e6090844ea814ab974f239e470]) successfully deleted
I20260927 00:39:28.046041 7898 catalog_manager.cc:4841] TS d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541): tablet 057b70cf06a94c9083cea17c8408c6c3 (table test-table [id=ea1fe8e6090844ea814ab974f239e470]) successfully deleted
I20260927 00:39:28.046522 7898 catalog_manager.cc:4841] TS d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593): tablet 057b70cf06a94c9083cea17c8408c6c3 (table test-table [id=ea1fe8e6090844ea814ab974f239e470]) successfully deleted
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:39:28.073980 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:28.067180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.98.62:36521)
0927 00:39:28.067323 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.067329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.067492 (+ 163us) server_negotiation.cc:407] Connection header received
0927 00:39:28.067618 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.067621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.067665 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.067748 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:28.068626 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.069218 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.070357 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.070568 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.070651 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.073846 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39118: BlockingRecv error: recv got EOF from 127.0.0.1:39118 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:39:28.085161 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.075913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.98.62:36521)
0927 00:39:28.076027 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.076033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.076234 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:39:28.076383 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.076386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.076434 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.076515 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:28.077126 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.077923 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.078797 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.079144 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.079258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.081833 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:28.081837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:28.081841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:28.081888 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:28.083463 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:28.083865 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:28.083869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:28.083871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:28.083955 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:28.084158 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:28.084161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:28.084163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:28.084418 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:28.084540 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.084873 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.084989 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:39:28.087711 7912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:28.088164 7912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:28.092896 8047 tablet_service.cc:1467] Processing CreateTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d (DEFAULT_TABLE table=test-table [id=11206c1c1a814f3f90206df64bb60b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.092949 8191 tablet_service.cc:1467] Processing CreateTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d (DEFAULT_TABLE table=test-table [id=11206c1c1a814f3f90206df64bb60b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.093165 8331 tablet_service.cc:1467] Processing CreateTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d (DEFAULT_TABLE table=test-table [id=11206c1c1a814f3f90206df64bb60b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.093226 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a465e0d47b814fb9bb9ea6772bff1a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.093225 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a465e0d47b814fb9bb9ea6772bff1a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.093397 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a465e0d47b814fb9bb9ea6772bff1a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.095273 8425 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Bootstrap starting.
I20260927 00:39:28.095561 8427 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Bootstrap starting.
I20260927 00:39:28.096131 8425 tablet_bootstrap.cc:654] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.096385 8427 tablet_bootstrap.cc:654] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.096735 8426 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Bootstrap starting.
I20260927 00:39:28.097352 8425 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: No bootstrap required, opened a new log
I20260927 00:39:28.097437 8425 ts_tablet_manager.cc:1397] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:28.097500 8426 tablet_bootstrap.cc:654] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.097663 8427 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: No bootstrap required, opened a new log
I20260927 00:39:28.097723 8427 ts_tablet_manager.cc:1397] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:28.097846 8425 raft_consensus.cc:348] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.097959 8425 raft_consensus.cc:374] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.097991 8425 raft_consensus.cc:729] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.098079 8427 raft_consensus.cc:348] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.098111 8425 consensus_queue.cc:260] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.098182 8427 raft_consensus.cc:374] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.098213 8427 raft_consensus.cc:729] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.098326 8427 consensus_queue.cc:260] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.098378 8425 ts_tablet_manager.cc:1428] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.098523 8426 tablet_bootstrap.cc:492] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: No bootstrap required, opened a new log
I20260927 00:39:28.098584 8427 ts_tablet_manager.cc:1428] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.098608 8426 ts_tablet_manager.cc:1397] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:39:28.098994 8426 raft_consensus.cc:348] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.099102 8426 raft_consensus.cc:374] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.099138 8426 raft_consensus.cc:729] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.099243 8426 consensus_queue.cc:260] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.099483 8426 ts_tablet_manager.cc:1428] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.296633 8433 raft_consensus.cc:482] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:28.296900 8433 raft_consensus.cc:504] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.297969 8433 leader_election.cc:288] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:28.302117 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.298831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48525 (local address 127.0.98.3:45593)
0927 00:39:28.298944 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.298948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.299029 (+ 81us) server_negotiation.cc:407] Connection header received
0927 00:39:28.299149 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.299152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.299198 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.299262 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:28.299510 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.300019 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301156 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.301568 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301671 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.301743 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.301907 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.301982 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:39:28.302140 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.298705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.98.2:48525)
0927 00:39:28.298957 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.298968 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:28.299053 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:28.299283 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:28.299286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:28.299300 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:28.299419 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.299424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.300123 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.300126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.301032 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.301042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.301684 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.301785 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.301809 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:39:28.301876 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:39:28.301957 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260927 00:39:28.302160 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.298368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41671 (local address 127.0.98.1:46541)
0927 00:39:28.298503 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.298508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.298734 (+ 226us) server_negotiation.cc:407] Connection header received
0927 00:39:28.298891 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.298894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.298952 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.299029 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:28.299415 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.299938 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301224 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.301637 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301734 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.301812 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.301963 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.302038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:39:28.302183 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.298359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.98.2:41671)
0927 00:39:28.298617 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.298630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:28.298765 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:28.299055 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:28.299059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:28.299078 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:28.299238 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.299248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.300062 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.300065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.301096 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.301106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.301748 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.301751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.301857 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.301877 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:39:28.301940 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:39:28.302015 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260927 00:39:28.302799 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a465e0d47b814fb9bb9ea6772bff1a0d" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c" is_pre_election: true
I20260927 00:39:28.302799 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a465e0d47b814fb9bb9ea6772bff1a0d" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" is_pre_election: true
I20260927 00:39:28.302968 8351 raft_consensus.cc:2454] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 0.
I20260927 00:39:28.302984 8071 raft_consensus.cc:2454] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 0.
I20260927 00:39:28.303320 8142 leader_election.cc:302] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:28.303603 8433 raft_consensus.cc:2781] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:28.303665 8433 raft_consensus.cc:482] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:28.303694 8433 raft_consensus.cc:3037] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.304776 8433 raft_consensus.cc:504] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.305130 8433 leader_election.cc:288] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 election: Requested vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:28.305569 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a465e0d47b814fb9bb9ea6772bff1a0d" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
I20260927 00:39:28.305585 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a465e0d47b814fb9bb9ea6772bff1a0d" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c"
I20260927 00:39:28.305698 8351 raft_consensus.cc:3037] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.305706 8071 raft_consensus.cc:3037] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.306695 8351 raft_consensus.cc:2454] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 1.
I20260927 00:39:28.306792 8071 raft_consensus.cc:2454] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 1.
I20260927 00:39:28.307015 8142 leader_election.cc:302] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d559eb6879274181ac611a0fdb10397c; no voters:
I20260927 00:39:28.307278 8433 raft_consensus.cc:2781] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:28.307488 8433 raft_consensus.cc:686] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 1 LEADER]: Becoming Leader. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Running, Role: LEADER
I20260927 00:39:28.307753 8433 consensus_queue.cc:237] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.310240 7913 catalog_manager.cc:5495] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add reported cstate change: term changed from 0 to 1, leader changed from <none> to 026504d0edef4ea081b3385d5f036add (127.0.98.2). New cstate: current_term: 1 leader_uuid: "026504d0edef4ea081b3385d5f036add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:28.337792 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.333799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.98.2:46801)
0927 00:39:28.333946 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.333949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.333965 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:39:28.334133 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.334136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.334232 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.334377 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:28.334605 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.335120 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.336055 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.336351 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.336434 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.336722 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.337326 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.337426 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.337564 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.337639 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:39:28.340689 8071 raft_consensus.cc:1261] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Refusing update from remote peer 026504d0edef4ea081b3385d5f036add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:28.340689 8351 raft_consensus.cc:1261] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Refusing update from remote peer 026504d0edef4ea081b3385d5f036add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:28.341311 8506 consensus_queue.cc:1035] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [LEADER]: Connected to new peer: Peer: permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:28.341581 8433 consensus_queue.cc:1035] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:28.352157 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.352363 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.354688 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Sending DeleteTablet for 3 replicas of tablet a465e0d47b814fb9bb9ea6772bff1a0d
I20260927 00:39:28.355361 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48750
I20260927 00:39:28.355434 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44690
I20260927 00:39:28.355564 8488 tablet_replica.cc:330] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: stopping tablet replica
I20260927 00:39:28.355633 8487 tablet_replica.cc:330] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: stopping tablet replica
I20260927 00:39:28.355683 8488 raft_consensus.cc:2229] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.355798 8487 raft_consensus.cc:2229] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.355820 8488 raft_consensus.cc:2258] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.355935 8487 raft_consensus.cc:2258] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.356555 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet a465e0d47b814fb9bb9ea6772bff1a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60458
I20260927 00:39:28.356720 8487 ts_tablet_manager.cc:1905] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.356768 8486 tablet_replica.cc:330] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: stopping tablet replica
I20260927 00:39:28.356861 8486 raft_consensus.cc:2229] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:28.356796 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:39:28.357208 8486 raft_consensus.cc:2258] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:39:28.357261 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:28.357971 8486 ts_tablet_manager.cc:1905] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.359719 8488 ts_tablet_manager.cc:1905] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.360414 8486 ts_tablet_manager.cc:1918] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:28.360494 8486 log.cc:1196] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/a465e0d47b814fb9bb9ea6772bff1a0d
I20260927 00:39:28.360724 8486 ts_tablet_manager.cc:1939] T a465e0d47b814fb9bb9ea6772bff1a0d P 026504d0edef4ea081b3385d5f036add: Deleting consensus metadata
I20260927 00:39:28.362023 8488 ts_tablet_manager.cc:1918] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:28.362104 8488 log.cc:1196] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/a465e0d47b814fb9bb9ea6772bff1a0d
I20260927 00:39:28.362330 8488 ts_tablet_manager.cc:1939] T a465e0d47b814fb9bb9ea6772bff1a0d P d7c76377be1749b2bafdf8b4b468e2d1: Deleting consensus metadata
I20260927 00:39:28.362293 7901 catalog_manager.cc:4841] TS 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801): tablet a465e0d47b814fb9bb9ea6772bff1a0d (table test-table [id=11206c1c1a814f3f90206df64bb60b5b]) successfully deleted
I20260927 00:39:28.362946 8487 ts_tablet_manager.cc:1918] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:28.363026 8487 log.cc:1196] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/a465e0d47b814fb9bb9ea6772bff1a0d
I20260927 00:39:28.363183 8487 ts_tablet_manager.cc:1939] T a465e0d47b814fb9bb9ea6772bff1a0d P d559eb6879274181ac611a0fdb10397c: Deleting consensus metadata
I20260927 00:39:28.363875 7898 catalog_manager.cc:4841] TS d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593): tablet a465e0d47b814fb9bb9ea6772bff1a0d (table test-table [id=11206c1c1a814f3f90206df64bb60b5b]) successfully deleted
I20260927 00:39:28.364581 7898 catalog_manager.cc:4841] TS d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541): tablet a465e0d47b814fb9bb9ea6772bff1a0d (table test-table [id=11206c1c1a814f3f90206df64bb60b5b]) successfully deleted
I20260927 00:39:28.366466 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 7274b3bbb292408e9273a64b76c8e367 (DEFAULT_TABLE table=test-table [id=07b7eabdff344eb9a10c7dcbc10f3edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.366621 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 7274b3bbb292408e9273a64b76c8e367 (DEFAULT_TABLE table=test-table [id=07b7eabdff344eb9a10c7dcbc10f3edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.366763 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274b3bbb292408e9273a64b76c8e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.366922 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274b3bbb292408e9273a64b76c8e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.368865 8427 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Bootstrap starting.
I20260927 00:39:28.369817 8427 tablet_bootstrap.cc:654] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.370642 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 7274b3bbb292408e9273a64b76c8e367 (DEFAULT_TABLE table=test-table [id=07b7eabdff344eb9a10c7dcbc10f3edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:28.370931 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7274b3bbb292408e9273a64b76c8e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:28.373448 8426 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Bootstrap starting.
I20260927 00:39:28.373912 8427 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: No bootstrap required, opened a new log
I20260927 00:39:28.374012 8427 ts_tablet_manager.cc:1397] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:39:28.374378 8426 tablet_bootstrap.cc:654] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.374553 8427 raft_consensus.cc:348] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.374701 8427 raft_consensus.cc:374] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.374732 8427 raft_consensus.cc:729] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.374851 8427 consensus_queue.cc:260] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.375108 8427 ts_tablet_manager.cc:1428] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.376276 8425 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: Bootstrap starting.
I20260927 00:39:28.377177 8425 tablet_bootstrap.cc:654] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:28.377188 8426 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: No bootstrap required, opened a new log
I20260927 00:39:28.377283 8426 ts_tablet_manager.cc:1397] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 00:39:28.377679 8426 raft_consensus.cc:348] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.377805 8426 raft_consensus.cc:374] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.377839 8426 raft_consensus.cc:729] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559eb6879274181ac611a0fdb10397c, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.377969 8426 consensus_queue.cc:260] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.378223 8426 ts_tablet_manager.cc:1428] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.379019 8425 tablet_bootstrap.cc:492] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: No bootstrap required, opened a new log
I20260927 00:39:28.379115 8425 ts_tablet_manager.cc:1397] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:39:28.379518 8425 raft_consensus.cc:348] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.379637 8425 raft_consensus.cc:374] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:28.379668 8425 raft_consensus.cc:729] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c76377be1749b2bafdf8b4b468e2d1, State: Initialized, Role: FOLLOWER
I20260927 00:39:28.379824 8425 consensus_queue.cc:260] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.380103 8425 ts_tablet_manager.cc:1428] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:39:28.640357 8511 raft_consensus.cc:482] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:28.640532 8511 raft_consensus.cc:504] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.640950 8511 leader_election.cc:288] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:28.641542 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274b3bbb292408e9273a64b76c8e367" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c" is_pre_election: true
I20260927 00:39:28.641542 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274b3bbb292408e9273a64b76c8e367" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" is_pre_election: true
I20260927 00:39:28.641712 8071 raft_consensus.cc:2454] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 0.
I20260927 00:39:28.641712 8351 raft_consensus.cc:2454] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 0.
I20260927 00:39:28.642103 8142 leader_election.cc:302] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:28.642375 8511 raft_consensus.cc:2781] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:28.642441 8511 raft_consensus.cc:482] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:28.642486 8511 raft_consensus.cc:3037] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.643610 8511 raft_consensus.cc:504] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.643949 8511 leader_election.cc:288] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 election: Requested vote from peers d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541), d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593)
I20260927 00:39:28.644557 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274b3bbb292408e9273a64b76c8e367" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c76377be1749b2bafdf8b4b468e2d1"
I20260927 00:39:28.644557 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7274b3bbb292408e9273a64b76c8e367" candidate_uuid: "026504d0edef4ea081b3385d5f036add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d559eb6879274181ac611a0fdb10397c"
I20260927 00:39:28.644695 8071 raft_consensus.cc:3037] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.644695 8351 raft_consensus.cc:3037] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:28.645823 8071 raft_consensus.cc:2454] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 1.
I20260927 00:39:28.645826 8351 raft_consensus.cc:2454] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026504d0edef4ea081b3385d5f036add in term 1.
I20260927 00:39:28.646145 8142 leader_election.cc:302] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026504d0edef4ea081b3385d5f036add, d7c76377be1749b2bafdf8b4b468e2d1; no voters:
I20260927 00:39:28.646311 8511 raft_consensus.cc:2781] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:28.646406 8511 raft_consensus.cc:686] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 1 LEADER]: Becoming Leader. State: Replica: 026504d0edef4ea081b3385d5f036add, State: Running, Role: LEADER
I20260927 00:39:28.646572 8511 consensus_queue.cc:237] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } }
I20260927 00:39:28.648397 7914 catalog_manager.cc:5495] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add reported cstate change: term changed from 0 to 1, leader changed from <none> to 026504d0edef4ea081b3385d5f036add (127.0.98.2). New cstate: current_term: 1 leader_uuid: "026504d0edef4ea081b3385d5f036add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026504d0edef4ea081b3385d5f036add" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 } health_report { overall_health: UNKNOWN } } }
I20260927 00:39:28.682953 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.679891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39167 (local address 127.0.98.3:45593)
0927 00:39:28.680029 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.680033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.680177 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:39:28.680339 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.680341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.680393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.680471 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:28.680706 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.681224 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.682068 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.682277 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.682358 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.682569 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.682637 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.682688 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.682760 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.682809 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:39:28.683037 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.679706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45593 (local address 127.0.98.2:39167)
0927 00:39:28.680074 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.680088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:28.680187 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:28.680479 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:28.680481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:28.680488 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:28.680583 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.680587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.681326 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.681329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.681960 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.681969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.682365 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.682367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.682424 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.682471 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.682693 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.682697 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:39:28.682747 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 00:39:28.682851 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260927 00:39:28.689298 8351 raft_consensus.cc:1261] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Refusing update from remote peer 026504d0edef4ea081b3385d5f036add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:28.689380 8071 raft_consensus.cc:1261] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Refusing update from remote peer 026504d0edef4ea081b3385d5f036add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:28.689744 8511 consensus_queue.cc:1035] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c76377be1749b2bafdf8b4b468e2d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:28.690027 8433 consensus_queue.cc:1035] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [LEADER]: Connected to new peer: Peer: permanent_uuid: "d559eb6879274181ac611a0fdb10397c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:28.708787 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.705539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.98.3:38243)
0927 00:39:28.705835 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.705847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:28.705945 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:28.706151 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:28.706154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:28.706161 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:39:28.706258 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.706264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.707021 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.707024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.707637 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:28.707646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.708099 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.708103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:28.708223 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.708252 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.708511 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.708517 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:28.708575 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 00:39:28.708661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260927 00:39:28.708784 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.705616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38243 (local address 127.0.98.2:46801)
0927 00:39:28.705722 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:28.705726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:28.705923 (+ 197us) server_negotiation.cc:407] Connection header received
0927 00:39:28.706026 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:28.706029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:28.706079 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:28.706136 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:28.706354 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.706912 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.707752 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:28.707985 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:28.708068 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:28.708363 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:28.708443 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:28.708500 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:28.708584 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 00:39:28.708638 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:39:28.731045 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.731218 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:28.733649 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f53c53ef43d43968b37dd0f452ba246: Sending DeleteTablet for 3 replicas of tablet 7274b3bbb292408e9273a64b76c8e367
I20260927 00:39:28.734582 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274b3bbb292408e9273a64b76c8e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60458
I20260927 00:39:28.734577 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274b3bbb292408e9273a64b76c8e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44690
I20260927 00:39:28.734783 8486 tablet_replica.cc:330] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: stopping tablet replica
I20260927 00:39:28.734753 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 7274b3bbb292408e9273a64b76c8e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48750
I20260927 00:39:28.734812 8487 tablet_replica.cc:330] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: stopping tablet replica
I20260927 00:39:28.734896 8486 raft_consensus.cc:2229] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:28.734941 8488 tablet_replica.cc:330] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1: stopping tablet replica
I20260927 00:39:28.734934 8487 raft_consensus.cc:2229] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.735059 8487 raft_consensus.cc:2258] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.735056 8488 raft_consensus.cc:2229] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:28.735114 8486 raft_consensus.cc:2258] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.735185 8488 raft_consensus.cc:2258] T 7274b3bbb292408e9273a64b76c8e367 P d7c76377be1749b2bafdf8b4b468e2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:28.735242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7986
I20260927 00:39:28.737437 8486 ts_tablet_manager.cc:1905] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.737839 8487 ts_tablet_manager.cc:1905] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:28.739914 8487 ts_tablet_manager.cc:1918] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:28.740000 8487 log.cc:1196] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/7274b3bbb292408e9273a64b76c8e367
I20260927 00:39:28.740250 8487 ts_tablet_manager.cc:1939] T 7274b3bbb292408e9273a64b76c8e367 P d559eb6879274181ac611a0fdb10397c: Deleting consensus metadata
I20260927 00:39:28.740314 8486 ts_tablet_manager.cc:1918] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:39:28.740404 8486 log.cc:1196] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/7274b3bbb292408e9273a64b76c8e367
I20260927 00:39:28.740662 8486 ts_tablet_manager.cc:1939] T 7274b3bbb292408e9273a64b76c8e367 P 026504d0edef4ea081b3385d5f036add: Deleting consensus metadata
I20260927 00:39:28.741091 7898 catalog_manager.cc:4841] TS d559eb6879274181ac611a0fdb10397c (127.0.98.3:45593): tablet 7274b3bbb292408e9273a64b76c8e367 (table test-table [id=07b7eabdff344eb9a10c7dcbc10f3edd]) successfully deleted
I20260927 00:39:28.741464 7901 catalog_manager.cc:4841] TS 026504d0edef4ea081b3385d5f036add (127.0.98.2:46801): tablet 7274b3bbb292408e9273a64b76c8e367 (table test-table [id=07b7eabdff344eb9a10c7dcbc10f3edd]) successfully deleted
W20260927 00:39:28.743534 7898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46541 (error 108)
I20260927 00:39:28.743734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8126
I20260927 00:39:28.745303 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:28.744887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.0.1:48776)
0927 00:39:28.745135 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:28.745209 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
W20260927 00:39:28.745466 7898 catalog_manager.cc:4584] TS d7c76377be1749b2bafdf8b4b468e2d1 (127.0.98.1:46541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7274b3bbb292408e9273a64b76c8e367: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46541: connect: Connection refused (error 111)
I20260927 00:39:28.750730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8266
I20260927 00:39:28.756711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 7882
2026-09-27T00:39:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3491 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[8532](info): set up 2 sockets
Sep 27 00:39:28 dist-test-slave-xltq krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260927 00:39:29.720834 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:29 dist-test-slave-xltq krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:29.846702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:29.958798 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:29.959100 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:29.959159 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:29.963325 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:29.963403 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:29.963431 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:29.963452 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:29.963471 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:29.969498 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:29.970675 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:29.971742 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:29.981432 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:29.981508 8545 server_base.cc:1034] running on GCE node
W20260927 00:39:29.981537 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:29.981537 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:29.982399 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:29.983609 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:29.984783 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790469569984713 us; error 84 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:39:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1238 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:30 dist-test-slave-xltq krb5kdc[8558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:30 dist-test-slave-xltq krb5kdc[8558](info): set up 2 sockets
Sep 27 00:39:30 dist-test-slave-xltq krb5kdc[8558](info): commencing operation
krb5kdc: starting...
W20260927 00:39:30.964805 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:31.086977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:31.201375 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:31.201691 8571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:31.201732 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:31.205969 8571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:31.206048 8571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:31.206066 8571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:31.206082 8571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:31.206102 8571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:31.211661 8571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:31.212797 8571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:31.213843 8571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:31.222518 8577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.222518 8576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.222664 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:31.222838 8571 server_base.cc:1034] running on GCE node
I20260927 00:39:31.223567 8571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:31.224869 8571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:31.226053 8571 hybrid_clock.cc:648] HybridClock initialized: now 1790469571226023 us; error 45 us; skew 500 ppm
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:31.229327 8571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:31.230304 8571 webserver.cc:460] Webserver started at http://127.0.98.62:43433/ using document root <none> and password file <none>
I20260927 00:39:31.230842 8571 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:31.230902 8571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:31.231112 8571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:31.233378 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "dc2e42f668b84e738d35aab05f8dff88"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "f6ea9c32a35e99305c5f7974c1474854"
server_key_iv: "0d6c718a408c5e34df52105674669ffb"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.233964 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc2e42f668b84e738d35aab05f8dff88"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "f6ea9c32a35e99305c5f7974c1474854"
server_key_iv: "0d6c718a408c5e34df52105674669ffb"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.238013 8571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:31.240854 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.242420 8571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:31.242566 8571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "dc2e42f668b84e738d35aab05f8dff88"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "f6ea9c32a35e99305c5f7974c1474854"
server_key_iv: "0d6c718a408c5e34df52105674669ffb"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.242697 8571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:31.264415 8571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:31.268036 8571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:31.268277 8571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:31.275329 8571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39563
I20260927 00:39:31.275375 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39563 every 8 connection(s)
I20260927 00:39:31.276346 8571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:31.279320 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:31.283963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8571
I20260927 00:39:31.284085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:31.284364 392 external_mini_cluster.cc:1428] Setting key dcc0b6188974b31a7675535eeb6d627e
I20260927 00:39:31.285246 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Bootstrap starting.
I20260927 00:39:31.288002 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:31.288913 8639 log.cc:824] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:31.291024 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: No bootstrap required, opened a new log
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:31.295070 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:31.295383 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:31.295444 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2e42f668b84e738d35aab05f8dff88, State: Initialized, Role: FOLLOWER
I20260927 00:39:31.295944 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:31.296077 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:31.296142 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:31.296276 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:31.297415 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:31.297772 8639 leader_election.cc:302] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2e42f668b84e738d35aab05f8dff88; no voters:
I20260927 00:39:31.298128 8639 leader_election.cc:288] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:31.298233 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:31.298445 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 LEADER]: Becoming Leader. State: Replica: dc2e42f668b84e738d35aab05f8dff88, State: Running, Role: LEADER
I20260927 00:39:31.298771 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:31.299484 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:31.299872 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:31.285825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.0.98.62:39563)
0927 00:39:31.286285 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:31.286315 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:31.286364 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:39:31.287220 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:31.287254 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:31.287689 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:31.288076 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:31.288834 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.290117 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.291026 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.291719 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.291880 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:31.294395 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:31.294402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:31.294423 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:31.294461 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:31.296319 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.296860 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.296866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.296872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.297000 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.297317 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.297320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.297321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.297798 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:31.297991 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:31.298420 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 00:39:31.298839 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260927 00:39:31.300704 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2e42f668b84e738d35aab05f8dff88. Latest consensus state: current_term: 1 leader_uuid: "dc2e42f668b84e738d35aab05f8dff88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 00:39:31.301298 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:31.301044 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc2e42f668b84e738d35aab05f8dff88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 00:39:31.301654 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: This master's current role is: LEADER
W20260927 00:39:31.304608 8659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:39:31.304708 8659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:39:31.304773 8658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:31.305634 8658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:31.311535 8658 catalog_manager.cc:1347] Generated new cluster ID: 0767b89421864610b75a824044630ec9
I20260927 00:39:31.311622 8658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:31.334838 8658 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:31.336347 8658 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:31.344365 8658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Generated new TSK 0
I20260927 00:39:31.345291 8658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:31.375525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:31.488034 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:31.488379 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:31.488426 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:31.492617 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:31.492688 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:31.492758 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:31.498659 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:31.499858 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:31.501056 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:31.511307 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.511303 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.511336 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:31.511761 8665 server_base.cc:1034] running on GCE node
I20260927 00:39:31.512137 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:31.512844 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:31.514068 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790469571513997 us; error 82 us; skew 500 ppm
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:31.517189 8665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:31.518222 8665 webserver.cc:460] Webserver started at http://127.0.98.1:42189/ using document root <none> and password file <none>
I20260927 00:39:31.518822 8665 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:31.518899 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:31.519124 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:31.521415 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "8022cfc2ce2d451f8e0edf03c2156c75"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "43c3971b910415c7ee961043ef68c15c"
server_key_iv: "5bad3f70e61d496cc6a3bcc4df10dcc8"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.522055 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8022cfc2ce2d451f8e0edf03c2156c75"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "43c3971b910415c7ee961043ef68c15c"
server_key_iv: "5bad3f70e61d496cc6a3bcc4df10dcc8"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.526268 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:31.529160 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.530776 8665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:31.530925 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "8022cfc2ce2d451f8e0edf03c2156c75"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "43c3971b910415c7ee961043ef68c15c"
server_key_iv: "5bad3f70e61d496cc6a3bcc4df10dcc8"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.531052 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:31.549358 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:31.553359 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:31.553529 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:31.554095 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:31.554949 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:31.554996 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.555047 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:31.555066 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.563628 8665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42795
I20260927 00:39:31.563658 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42795 every 8 connection(s)
I20260927 00:39:31.564607 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:31.572774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8665
I20260927 00:39:31.572904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:31.573164 392 external_mini_cluster.cc:1428] Setting key 69e9bd31bb2e3fedc4bc3a69c542eb76
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:31.579893 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:31.566613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48279 (local address 127.0.98.62:39563)
0927 00:39:31.566803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:31.566807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:31.567699 (+ 892us) server_negotiation.cc:407] Connection header received
0927 00:39:31.569323 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:31.569327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:31.569381 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:31.569453 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:31.570712 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.571241 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.572529 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.572763 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.572855 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:31.575667 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:31.575713 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:31.575716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:31.575743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:31.577406 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.578075 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.578078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.578079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.578165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.578591 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.578595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.578596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.578815 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:31.578907 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:31.579636 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 00:39:31.579764 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 00:39:31.581197 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:31.567032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:48279)
0927 00:39:31.567535 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:31.567583 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:31.569014 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:31.569646 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:31.569655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:31.570043 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:31.570557 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:31.570571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.571366 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.571369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:31.572385 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:31.572396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.572915 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.572918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:31.573093 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:31.573773 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:31.573795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:31.575491 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:31.577548 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:31.577555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:31.577576 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:31.577939 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:31.578270 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:31.578273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:31.578275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:31.578470 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:31.578933 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:31.578940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:31.579331 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 00:39:31.579567 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 00:39:31.579981 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260927 00:39:31.582793 8794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 00:39:31.583063 8794 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:39:31.583691 8794 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 00:39:31.585439 8603 ts_manager.cc:194] Registered new tserver with Master: 8022cfc2ce2d451f8e0edf03c2156c75 (127.0.98.1:42795)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:39:31.587031 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:31.603984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:31.716636 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:31.716951 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:31.717015 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:31.721400 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:31.721479 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:31.721565 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:31.727458 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:31.728755 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:31.729866 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:31.740386 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:31.740543 8799 server_base.cc:1034] running on GCE node
W20260927 00:39:31.740374 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.740662 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:31.741113 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:31.741853 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:31.743068 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790469571743005 us; error 76 us; skew 500 ppm
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:31.746137 8799 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:31.747150 8799 webserver.cc:460] Webserver started at http://127.0.98.2:39789/ using document root <none> and password file <none>
I20260927 00:39:31.747740 8799 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:31.747815 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:31.748013 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:31.750406 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "22d34a6eb7e047b9a1acfa981a3decd2"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "1b09c235c099775806372fa8d6450add"
server_key_iv: "672f9c681e7e159afefd6b02b10c2776"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.751019 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "22d34a6eb7e047b9a1acfa981a3decd2"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "1b09c235c099775806372fa8d6450add"
server_key_iv: "672f9c681e7e159afefd6b02b10c2776"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.755244 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:31.758006 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.759600 8799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:31.759753 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "22d34a6eb7e047b9a1acfa981a3decd2"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "1b09c235c099775806372fa8d6450add"
server_key_iv: "672f9c681e7e159afefd6b02b10c2776"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.759879 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:31.773662 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:31.777694 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:31.777892 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:31.778491 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:31.779700 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:31.779804 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.779855 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:31.779876 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.789000 8799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39749
I20260927 00:39:31.789034 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39749 every 8 connection(s)
I20260927 00:39:31.789917 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:31.790752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8799
I20260927 00:39:31.790864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:31.791118 392 external_mini_cluster.cc:1428] Setting key 3123e81feab35d722c1d0582fc6f20f7
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:31.809857 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:31.791931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49597 (local address 127.0.98.62:39563)
0927 00:39:31.792059 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:31.792064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:31.793109 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 00:39:31.794826 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:31.794829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:31.794887 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:31.794971 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:31.796525 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.797100 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.798898 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.799219 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.799325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:31.805913 (+ 6588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:31.805916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:31.805919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:31.805950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:31.807507 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.808152 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.808155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.808156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.808287 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:31.808664 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:31.808667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:31.808669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:31.808912 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:31.809011 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:31.809557 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 00:39:31.809690 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:39:31.810722 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:31.792432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:49597)
0927 00:39:31.792923 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:31.792983 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:31.794597 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:31.795177 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:31.795185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:31.795653 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:31.796360 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:31.796381 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.797242 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.797246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:31.798745 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:31.798757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:31.799404 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:31.799408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:31.799636 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:31.800479 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:31.800504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:31.805633 (+ 5129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:31.807676 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:31.807690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:31.807707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:31.808010 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:31.808382 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:31.808385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:31.808387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:31.808555 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:31.809011 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:31.809019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:31.809309 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 00:39:31.809481 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 00:39:31.809859 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260927 00:39:31.811923 8928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 00:39:31.812242 8928 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:31.812777 8928 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 00:39:31.813799 8603 ts_manager.cc:194] Registered new tserver with Master: 22d34a6eb7e047b9a1acfa981a3decd2 (127.0.98.2:39749)
I20260927 00:39:31.814450 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:31.823804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:31.937222 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:31.937481 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:31.937525 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:31.941679 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:31.941751 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:31.941815 8933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:31.947415 8933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:31.948602 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:31.949651 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:31.959656 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.959657 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:31.959657 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:31.959724 8933 server_base.cc:1034] running on GCE node
I20260927 00:39:31.960381 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:31.961102 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:31.962291 8933 hybrid_clock.cc:648] HybridClock initialized: now 1790469571962255 us; error 51 us; skew 500 ppm
Sep 27 00:39:31 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:31.965384 8933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:31.966259 8933 webserver.cc:460] Webserver started at http://127.0.98.3:41401/ using document root <none> and password file <none>
I20260927 00:39:31.966789 8933 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:31.966847 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:31.967028 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:31.969209 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "44e4b1e96ee34fcbbe83c0c58ce4d4b4"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "360d7012dc55b1c6456968334d5a2a02"
server_key_iv: "c9bca623f0f972f744c10d69ebf0dd09"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.969774 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "44e4b1e96ee34fcbbe83c0c58ce4d4b4"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "360d7012dc55b1c6456968334d5a2a02"
server_key_iv: "c9bca623f0f972f744c10d69ebf0dd09"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.973587 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:31.976382 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.977846 8933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:31.977964 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "44e4b1e96ee34fcbbe83c0c58ce4d4b4"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "360d7012dc55b1c6456968334d5a2a02"
server_key_iv: "c9bca623f0f972f744c10d69ebf0dd09"
server_key_version: "encryptionkey@0"
I20260927 00:39:31.978106 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:31.991946 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:31.995550 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:31.995712 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:31.996357 8933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:31.997185 8933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:31.997241 8933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:31.997282 8933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:31.997303 8933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.006132 8933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44299
I20260927 00:39:32.006212 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44299 every 8 connection(s)
I20260927 00:39:32.007078 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:32.010541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 8933
I20260927 00:39:32.010671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:32.010917 392 external_mini_cluster.cc:1428] Setting key 1c275a38f67f9bec6f43421967700028
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:32.021119 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.008938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49399 (local address 127.0.98.62:39563)
0927 00:39:32.009063 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.009066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.009941 (+ 875us) server_negotiation.cc:407] Connection header received
0927 00:39:32.011428 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.011432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.011489 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.011563 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.012805 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.013318 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.014519 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.014734 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.014829 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.017399 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:32.017402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:32.017404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:32.017430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:32.018937 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:32.019526 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:32.019529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:32.019530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:32.019608 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:32.019977 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:32.019979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:32.019980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:32.020170 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:32.020301 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:32.020889 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 00:39:32.020999 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:39:32.021965 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.009331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:49399)
0927 00:39:32.009782 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.009828 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.011193 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.011759 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.011766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.012132 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.012665 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.012680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.013434 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.013437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.014402 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.014412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.014873 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.014877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.015050 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.015596 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.015613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.017140 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:32.019063 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:32.019069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:32.019088 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:32.019427 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:32.019718 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:32.019721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:32.019722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:32.019891 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:32.020312 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:32.020319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:32.020587 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 00:39:32.020813 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 00:39:32.021122 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260927 00:39:32.023125 9062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 00:39:32.023398 9062 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:32.023993 9062 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 00:39:32.025099 8603 ts_manager.cc:194] Registered new tserver with Master: 44e4b1e96ee34fcbbe83c0c58ce4d4b4 (127.0.98.3:44299)
I20260927 00:39:32.025640 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49399
I20260927 00:39:32.036588 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:39:32.111657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8665
I20260927 00:39:32.116482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8799
I20260927 00:39:32.121320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8933
I20260927 00:39:32.126302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 8571
I20260927 00:39:32.132107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--webserver_interface=127.0.98.62
--webserver_port=43433
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:32.243561 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:32.243868 9071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:32.243911 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:32.248067 9071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:32.248140 9071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:32.248158 9071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:32.248174 9071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:32.248189 9071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:32.254101 9071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:32.255177 9071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:32.256181 9071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:32.265132 9076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.265189 9071 server_base.cc:1034] running on GCE node
W20260927 00:39:32.265115 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:32.265115 9077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.265947 9071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:32.267165 9071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:32.268296 9071 hybrid_clock.cc:648] HybridClock initialized: now 1790469572268249 us; error 58 us; skew 500 ppm
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:32.271200 9071 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:32.272270 9071 webserver.cc:460] Webserver started at http://127.0.98.62:43433/ using document root <none> and password file <none>
I20260927 00:39:32.272830 9071 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:32.272913 9071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:32.277087 9071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:39:32.279422 9086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.280761 9071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:39:32.280928 9071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "dc2e42f668b84e738d35aab05f8dff88"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "f6ea9c32a35e99305c5f7974c1474854"
server_key_iv: "0d6c718a408c5e34df52105674669ffb"
server_key_version: "encryptionkey@0"
I20260927 00:39:32.281487 9071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:32.294745 9071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:32.298019 9071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:32.298209 9071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:32.305413 9071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39563
I20260927 00:39:32.305426 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39563 every 8 connection(s)
I20260927 00:39:32.306530 9071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:32.308691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9071
I20260927 00:39:32.309146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42795
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39563
--webserver_port=42189
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:39:32.311569 9139 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:39:32.313434 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Bootstrap starting.
I20260927 00:39:32.329998 9139 log.cc:824] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:32.336781 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:39:32.337173 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Bootstrap complete.
I20260927 00:39:32.341344 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:32.341784 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2e42f668b84e738d35aab05f8dff88, State: Initialized, Role: FOLLOWER
I20260927 00:39:32.342274 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:32.342398 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:32.342466 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:32.342558 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:39:32.344693 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:32.345064 9139 leader_election.cc:302] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc2e42f668b84e738d35aab05f8dff88; no voters:
I20260927 00:39:32.345407 9139 leader_election.cc:288] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:39:32.345497 9143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:39:32.345685 9143 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [term 2 LEADER]: Becoming Leader. State: Replica: dc2e42f668b84e738d35aab05f8dff88, State: Running, Role: LEADER
I20260927 00:39:32.345981 9143 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 00:39:32.346688 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:32.347350 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc2e42f668b84e738d35aab05f8dff88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 00:39:32.347478 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:32.348090 9151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:32.348129 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc2e42f668b84e738d35aab05f8dff88. Latest consensus state: current_term: 2 leader_uuid: "dc2e42f668b84e738d35aab05f8dff88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc2e42f668b84e738d35aab05f8dff88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 00:39:32.348268 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:32.351493 9151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:32.352452 9151 catalog_manager.cc:1259] Loaded cluster ID: 0767b89421864610b75a824044630ec9
I20260927 00:39:32.352514 9151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:32.354919 9151 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:32.355578 9151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc2e42f668b84e738d35aab05f8dff88: Loaded TSK: 0
I20260927 00:39:32.356838 9151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:39:32.425755 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:32.426038 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:32.426080 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:32.430342 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:32.430418 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:32.430482 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:32.436075 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42189
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:32.437143 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:32.438236 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:32.448019 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.448226 9141 server_base.cc:1034] running on GCE node
W20260927 00:39:32.448056 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:32.448022 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.448860 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:32.449570 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:32.450737 9141 hybrid_clock.cc:648] HybridClock initialized: now 1790469572450701 us; error 55 us; skew 500 ppm
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469572, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:32.453831 9141 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:39:32.454861 9141 webserver.cc:460] Webserver started at http://127.0.98.1:42189/ using document root <none> and password file <none>
I20260927 00:39:32.455436 9141 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:32.455523 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:32.459600 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:32.462114 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.463551 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:32.463673 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "8022cfc2ce2d451f8e0edf03c2156c75"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "43c3971b910415c7ee961043ef68c15c"
server_key_iv: "5bad3f70e61d496cc6a3bcc4df10dcc8"
server_key_version: "encryptionkey@0"
I20260927 00:39:32.464324 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:32.496341 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:32.499966 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:32.500192 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:32.500901 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:32.501821 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:32.501874 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.501921 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:32.501956 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.511560 9141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42795
I20260927 00:39:32.511623 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42795 every 8 connection(s)
I20260927 00:39:32.512641 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:32.516392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9141
I20260927 00:39:32.517046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39563
--webserver_port=39789
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.527619 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.514933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:34967)
0927 00:39:32.515502 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.515566 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.517320 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.518884 (+ 1564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.518894 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.519351 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.519863 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.519879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.521288 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.521291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.522229 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.522240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.523045 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.523049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.523231 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.523818 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.523836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.526504 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
W20260927 00:39:32.527611 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.514855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34967 (local address 127.0.98.62:39563)
0927 00:39:32.515498 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.515527 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.515691 (+ 164us) server_negotiation.cc:407] Connection header received
0927 00:39:32.517910 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.517923 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.518280 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.518753 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.520056 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.521136 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.522371 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.522870 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.523033 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.526785 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34967: BlockingRecv error: recv got EOF from 127.0.98.1:34967 (error 108)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":125,"threads_started":1}
W20260927 00:39:32.528141 9289 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39563 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39563: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.534003 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.528634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:46281)
0927 00:39:32.528793 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.528850 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.528941 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.529176 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.529179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.529344 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.529463 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.529471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.530253 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.530257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.530839 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.530849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.531295 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.531299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.531397 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.531857 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.531869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.533855 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 00:39:32.534179 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.528745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46281 (local address 127.0.98.62:39563)
0927 00:39:32.528868 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.528872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.528891 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:39:32.529028 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.529031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.529084 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.529148 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.529598 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.530141 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.530970 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.531188 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.531278 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.534072 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46281: BlockingRecv error: recv got EOF from 127.0.98.1:46281 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.540325 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.534620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:51951)
0927 00:39:32.534719 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.534733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.534808 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.535083 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.535086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.535235 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.535410 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.535415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.536476 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.536479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.537046 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.537056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.537647 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.537650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.537719 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.538212 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.538225 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.540165 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 00:39:32.540521 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.534707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51951 (local address 127.0.98.62:39563)
0927 00:39:32.534826 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.534830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.534846 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:39:32.534891 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.534894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.534942 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.535028 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.535522 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.536331 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.537181 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.537505 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.537609 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.540394 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51951: BlockingRecv error: recv got EOF from 127.0.98.1:51951 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:39:32.540623 9289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:39:32.633628 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:32.633951 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:32.633999 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:32.638270 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:32.638348 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:32.638408 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:32.644050 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39789
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:32.645216 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:32.646253 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:32.656394 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.656603 9293 server_base.cc:1034] running on GCE node
W20260927 00:39:32.656394 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:32.656394 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.657116 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:32.657869 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:32.659097 9293 hybrid_clock.cc:648] HybridClock initialized: now 1790469572659067 us; error 43 us; skew 500 ppm
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469572, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:32.662281 9293 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:39:32.663411 9293 webserver.cc:460] Webserver started at http://127.0.98.2:39789/ using document root <none> and password file <none>
I20260927 00:39:32.663985 9293 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:32.664070 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:32.668525 9293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:32.670953 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.672585 9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:32.672711 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "22d34a6eb7e047b9a1acfa981a3decd2"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "1b09c235c099775806372fa8d6450add"
server_key_iv: "672f9c681e7e159afefd6b02b10c2776"
server_key_version: "encryptionkey@0"
I20260927 00:39:32.673321 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:32.694986 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:32.698305 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:32.698494 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:32.699097 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:32.699939 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:32.699986 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.700039 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:32.700067 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.709231 9293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39749
I20260927 00:39:32.709261 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39749 every 8 connection(s)
I20260927 00:39:32.710273 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:32.713905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9293
I20260927 00:39:32.714403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44299
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39563
--webserver_port=41401
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:39:32.723263 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.712131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54027 (local address 127.0.98.62:39563)
0927 00:39:32.712325 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.712329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.713211 (+ 882us) server_negotiation.cc:407] Connection header received
0927 00:39:32.714850 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.714853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.714916 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.714986 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.716525 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.717047 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.718686 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.718927 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.719036 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.723132 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54027: BlockingRecv error: recv got EOF from 127.0.98.2:54027 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 00:39:32.724009 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.712584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:54027)
0927 00:39:32.713062 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.713105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.714593 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.715197 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.715205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.715703 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.716351 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.716369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.717178 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.717182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.718511 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.718524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.719119 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.719124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.719343 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.720030 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.720053 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.722979 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
W20260927 00:39:32.724668 9422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39563 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39563: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.730468 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.725181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:39917)
0927 00:39:32.725343 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.725359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.725601 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.725851 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.725854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.726026 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.726130 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.726136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.726936 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.726940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.727501 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.727510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.727963 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.727967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.728055 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.728568 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.728581 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.730323 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 00:39:32.730583 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.725444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39917 (local address 127.0.98.62:39563)
0927 00:39:32.725549 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.725554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.725567 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:32.725709 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.725712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.725754 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.725821 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.726252 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.726824 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.727626 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.727856 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.727948 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.730483 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39917: BlockingRecv error: recv got EOF from 127.0.98.2:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.737092 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.731202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:39323)
0927 00:39:32.731369 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.731382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.731455 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.731699 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.731703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.731883 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.731979 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.731985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.732968 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.732972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.733852 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.733863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.734329 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.734332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.734416 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.734925 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.734940 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.736897 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 00:39:32.737156 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.731294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39323 (local address 127.0.98.62:39563)
0927 00:39:32.731472 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.731476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.731490 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:32.731526 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.731528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.731566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.731630 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.732110 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.732853 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.733995 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.734218 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.734308 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.737056 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39323: BlockingRecv error: recv got EOF from 127.0.98.2:39323 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260927 00:39:32.737406 9422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:39:32.831925 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:32.832235 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:32.832296 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:32.836606 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:32.836683 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:32.836745 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:32.842305 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41401
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:32.843451 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:32.844564 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:32.854596 9430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:32.854736 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:32.854607 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:32.854907 9425 server_base.cc:1034] running on GCE node
I20260927 00:39:32.855418 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:32.856139 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:32.857329 9425 hybrid_clock.cc:648] HybridClock initialized: now 1790469572857308 us; error 42 us; skew 500 ppm
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469572, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:32.860173 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:39:32.861156 9425 webserver.cc:460] Webserver started at http://127.0.98.3:41401/ using document root <none> and password file <none>
I20260927 00:39:32.861742 9425 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:32.861850 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:32.866062 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:39:32.868470 9440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.869987 9425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:39:32.870100 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "44e4b1e96ee34fcbbe83c0c58ce4d4b4"
format_stamp: "Formatted at 2026-09-27 00:39:31 on dist-test-slave-xltq"
server_key: "360d7012dc55b1c6456968334d5a2a02"
server_key_iv: "c9bca623f0f972f744c10d69ebf0dd09"
server_key_version: "encryptionkey@0"
I20260927 00:39:32.870716 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:32.883792 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:32.887120 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:32.887283 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:32.887880 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:32.888753 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:32.888803 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.888842 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:32.888855 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:32.897557 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44299
I20260927 00:39:32.897650 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44299 every 8 connection(s)
I20260927 00:39:32.898592 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:32.901965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9425
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:39:32.911032 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.900485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55391 (local address 127.0.98.62:39563)
0927 00:39:32.900605 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.900608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.901595 (+ 987us) server_negotiation.cc:407] Connection header received
0927 00:39:32.903100 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.903103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.903155 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.903230 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.904647 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.905268 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.906788 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.906992 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.907103 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.910912 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55391: BlockingRecv error: recv got EOF from 127.0.98.3:55391 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:39:32.911722 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.900936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:55391)
0927 00:39:32.901380 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.901474 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.902866 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.903377 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.903387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.903867 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.904503 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.904517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.905395 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.905398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.906653 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.906665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.907171 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.907176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.907358 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.907953 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.907974 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.910814 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260927 00:39:32.912124 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.902969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.98.62:39563)
0927 00:39:32.903238 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.903242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.903253 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:39:32.903280 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.903282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.903312 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.903390 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.904268 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.904980 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.906124 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.906374 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.906459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.907925 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:32.907933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:32.907954 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:32.907991 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:32.909917 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:32.910385 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:32.910393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:32.910401 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:32.910567 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:32.910837 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:32.910840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:32.910841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:32.911277 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:32.911490 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:32.911745 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:39:32.911996 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":68,"threads_started":1}
W20260927 00:39:32.912221 9554 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39563 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39563: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.918028 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.912640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:46669)
0927 00:39:32.912845 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.912856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.912928 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.913142 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.913145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.913302 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.913420 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.913427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.914288 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.914292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.915070 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.915082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.915560 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.915565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.915656 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.916134 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.916151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.917921 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 00:39:32.918175 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.912724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46669 (local address 127.0.98.62:39563)
0927 00:39:32.912838 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.912842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.912893 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:39:32.912987 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.912989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.913031 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.913108 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.913524 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.914158 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.915187 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.915442 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.915534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.918055 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46669: BlockingRecv error: recv got EOF from 127.0.98.3:46669 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:32 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:32.923383 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:32.918595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:52153)
0927 00:39:32.918710 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:32.918724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:32.918803 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:32.919001 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:32.919003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:32.919126 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:32.919203 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.919207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.919893 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.919896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.920498 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:32.920506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.920941 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.920944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:32.921009 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.921508 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:32.921519 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:32.923277 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:39:32.923548 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:32.918679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52153 (local address 127.0.98.62:39563)
0927 00:39:32.918830 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:32.918833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:32.918844 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:39:32.918877 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:32.918880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:32.918914 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:32.918977 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:32.919314 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.919799 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.920603 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:32.920841 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:32.920929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:32.923433 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52153: BlockingRecv error: recv got EOF from 127.0.98.3:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 00:39:32.923633 9554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:33.548593 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:33.541563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:56765)
0927 00:39:33.541907 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:33.541923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:33.542044 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:33.542390 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:33.542393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:33.542656 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:33.542811 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.542832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.543907 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.543928 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.544699 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.544713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.545231 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.545236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.545351 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.545926 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:33.545938 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:33.548359 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260927 00:39:33.548874 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:33.541743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56765 (local address 127.0.98.62:39563)
0927 00:39:33.542024 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:33.542028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:33.542046 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:33.542148 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:33.542150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:33.542197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:33.542330 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:33.542998 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.543778 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.544832 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.545106 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.545209 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.548628 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56765: BlockingRecv error: recv got EOF from 127.0.98.1:56765 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:33.744611 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:33.738215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:39175)
0927 00:39:33.738492 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:33.738508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:33.738611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:33.738921 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:33.738924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:33.739137 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:33.739271 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.739281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.740278 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.740282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.740891 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.740904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.741473 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.741476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.741562 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.742103 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:33.742115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:33.744384 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
W20260927 00:39:33.744746 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:33.738412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39175 (local address 127.0.98.62:39563)
0927 00:39:33.738632 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:33.738636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:33.738656 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:39:33.738713 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:33.738715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:33.738769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:33.738891 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:33.739455 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.740091 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.741072 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.741340 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.741447 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.744627 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39175: BlockingRecv error: recv got EOF from 127.0.98.2:39175 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:33 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:33.931121 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:33.924433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:38061)
0927 00:39:33.924786 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:33.924805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:33.924958 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:33.925229 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:33.925233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:33.925519 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:33.925657 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.925664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.926497 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.926501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.927131 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:33.927142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.927689 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.927693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:33.927794 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.928462 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:33.928474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:33.930878 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
W20260927 00:39:33.931247 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:33.924544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38061 (local address 127.0.98.62:39563)
0927 00:39:33.924744 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:33.924750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:33.924892 (+ 142us) server_negotiation.cc:407] Connection header received
0927 00:39:33.925057 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:33.925061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:33.925116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:33.925187 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:33.925798 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.926385 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.927281 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:33.927569 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:33.927665 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:33.931118 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38061: BlockingRecv error: recv got EOF from 127.0.98.3:38061 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:34.556460 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:34.549844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:42135)
0927 00:39:34.550257 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:34.550275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:34.550441 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:34.550887 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:34.550890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:34.551127 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:34.551264 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.551271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.552170 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.552174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.552873 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.552884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.553371 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.553375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.553454 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.554045 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:34.554057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:34.556289 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
W20260927 00:39:34.556835 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:34.549987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42135 (local address 127.0.98.62:39563)
0927 00:39:34.550296 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:34.550300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:34.550443 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:39:34.550530 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:34.550532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:34.550682 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:34.550814 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:34.551416 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.552031 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.553001 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.553258 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.553347 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.556653 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42135: BlockingRecv error: recv got EOF from 127.0.98.1:42135 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:34.753544 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:34.745898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:37029)
0927 00:39:34.746402 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:34.746423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:34.746588 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:34.746919 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:34.746926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:34.747203 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:34.747356 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.747365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.748410 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.748415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.749166 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.749179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.749680 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.749683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.749766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.750463 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:34.750480 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:34.753245 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":105,"threads_started":1}
W20260927 00:39:34.753569 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:34.746005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37029 (local address 127.0.98.62:39563)
0927 00:39:34.746143 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:34.746147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:34.746594 (+ 447us) server_negotiation.cc:407] Connection header received
0927 00:39:34.746665 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:34.746667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:34.746724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:34.746820 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:34.747527 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.748256 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.749305 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.749539 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.749641 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.753425 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37029: BlockingRecv error: recv got EOF from 127.0.98.2:37029 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:34 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:34.939815 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:34.932280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:38043)
0927 00:39:34.932620 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:34.932638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:34.932761 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:34.933108 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:34.933111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:34.933330 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:34.933467 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.933473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.934406 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.934411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.935325 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:34.935350 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.936026 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.936031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:34.936136 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.936818 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:34.936838 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:34.939599 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
W20260927 00:39:34.939957 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:34.932537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38043 (local address 127.0.98.62:39563)
0927 00:39:34.932747 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:34.932751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:34.932767 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:39:34.932846 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:34.932848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:34.932911 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:34.933002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:34.933675 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.934220 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.935524 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:34.935829 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:34.935937 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:34.939812 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38043: BlockingRecv error: recv got EOF from 127.0.98.3:38043 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:35.563015 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:35.557339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:56355)
0927 00:39:35.557645 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:35.557659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:35.557761 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:35.557997 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:35.557999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:35.558173 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:35.558314 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.558322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.559184 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.559187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.559754 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.559763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.560346 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.560349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.560450 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.561031 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:35.561044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:35.562891 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
W20260927 00:39:35.563301 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:35.557480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56355 (local address 127.0.98.62:39563)
0927 00:39:35.557773 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:35.557777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:35.557792 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:35.557842 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:35.557844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:35.557890 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:35.557969 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:35.558449 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.559050 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.559885 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.560160 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.560326 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.563133 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56355: BlockingRecv error: recv got EOF from 127.0.98.1:56355 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:35.761693 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:35.754501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:50595)
0927 00:39:35.754908 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:35.754925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:35.755067 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:35.755349 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:35.755352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:35.755583 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:35.755724 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.755732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.756856 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.756860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.757534 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.757546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.758209 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.758213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.758335 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.759009 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:35.759022 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:35.761485 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":82,"threads_started":1}
W20260927 00:39:35.761788 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:35.754651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50595 (local address 127.0.98.62:39563)
0927 00:39:35.754804 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:35.754808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:35.755020 (+ 212us) server_negotiation.cc:407] Connection header received
0927 00:39:35.755166 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:35.755168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:35.755222 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:35.755308 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:35.755947 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.756734 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.757693 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.758085 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.758195 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.761685 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50595: BlockingRecv error: recv got EOF from 127.0.98.2:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:35 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:35.950922 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:35.940665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:42343)
0927 00:39:35.940954 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:35.940967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:35.941082 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:35.941321 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:35.941324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:35.941524 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:35.941621 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.941627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.942409 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.942413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.942976 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:35.942984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.943402 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.943406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:35.943487 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.948567 (+ 5080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:35.948582 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:35.950789 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260927 00:39:35.951079 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:35.940742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42343 (local address 127.0.98.62:39563)
0927 00:39:35.940904 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:35.940909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:35.941038 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:39:35.941164 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:35.941166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:35.941213 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:35.941290 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:35.941782 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.942291 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.943085 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:35.943301 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:35.943385 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:35.950959 (+ 7574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42343: BlockingRecv error: recv got EOF from 127.0.98.3:42343 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:36.569819 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:36.563911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:57471)
0927 00:39:36.564187 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:36.564246 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:36.564379 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:36.564649 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:36.564652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:36.564913 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:36.565013 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.565019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.565897 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.565900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.566475 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.566484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.567042 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.567045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.567113 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.567646 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:36.567658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:36.569674 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
W20260927 00:39:36.570051 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:36.563984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57471 (local address 127.0.98.62:39563)
0927 00:39:36.564257 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:36.564262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:36.564318 (+ 56us) server_negotiation.cc:407] Connection header received
0927 00:39:36.564454 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:36.564456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:36.564500 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:36.564578 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:36.565162 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.565706 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.566608 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.566852 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.567017 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.569839 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57471: BlockingRecv error: recv got EOF from 127.0.98.1:57471 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:36.769120 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:36.762790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:45007)
0927 00:39:36.763084 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:36.763097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:36.763191 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:36.763495 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:36.763498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:36.763668 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:36.763768 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.763773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.764694 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.764697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.765373 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.765384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.765988 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.765991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.766064 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.766697 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:36.766713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:36.768952 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
W20260927 00:39:36.769243 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:36.762928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45007 (local address 127.0.98.62:39563)
0927 00:39:36.763139 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:36.763142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:36.763160 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:39:36.763314 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:36.763317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:36.763366 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:36.763454 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:36.763910 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.764516 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.765536 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.765822 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.765936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.769139 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45007: BlockingRecv error: recv got EOF from 127.0.98.2:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:36 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:36.957849 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:36.951817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:55185)
0927 00:39:36.952139 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:36.952156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:36.952319 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:36.952605 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:36.952607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:36.952776 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:36.952870 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.952875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.953649 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.953652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.954208 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:36.954216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.954738 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.954741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:36.954808 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.955474 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:36.955487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:36.957721 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":77,"threads_started":1}
W20260927 00:39:36.958063 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:36.951928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55185 (local address 127.0.98.62:39563)
0927 00:39:36.952131 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:36.952136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:36.952234 (+ 98us) server_negotiation.cc:407] Connection header received
0927 00:39:36.952431 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:36.952435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:36.952489 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:36.952557 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:36.953010 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.953542 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.954334 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:36.954593 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:36.954686 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:36.957877 (+ 3191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55185: BlockingRecv error: recv got EOF from 127.0.98.3:55185 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:37.577255 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:37.570651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:48523)
0927 00:39:37.570903 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:37.570919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:37.571046 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:37.571329 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:37.571332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:37.571548 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:37.571665 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.571672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.572723 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.572727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.573624 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.573638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.574233 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.574237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.574306 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.574942 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:37.574958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:37.577086 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
W20260927 00:39:37.577363 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:37.570691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48523 (local address 127.0.98.62:39563)
0927 00:39:37.570905 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:37.570909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:37.570988 (+ 79us) server_negotiation.cc:407] Connection header received
0927 00:39:37.571167 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:37.571170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:37.571222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:37.571308 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:37.571765 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.572614 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.573755 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.574127 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.574222 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.577240 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48523: BlockingRecv error: recv got EOF from 127.0.98.1:48523 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:37.776292 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:37.770234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:53041)
0927 00:39:37.770480 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:37.770494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:37.770626 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:37.770914 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:37.770917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:37.771132 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:37.771227 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.771233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.771999 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.772002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.772600 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.772610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.773168 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.773171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.773248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.773841 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:37.773858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:37.776097 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260927 00:39:37.776438 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:37.770381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53041 (local address 127.0.98.62:39563)
0927 00:39:37.770514 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:37.770519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:37.770579 (+ 60us) server_negotiation.cc:407] Connection header received
0927 00:39:37.770719 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:37.770723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:37.770773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:37.770872 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:37.771353 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.771893 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.772712 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.773064 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.773160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.776314 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53041: BlockingRecv error: recv got EOF from 127.0.98.2:53041 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:37 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:37.964186 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:37.958751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:51795)
0927 00:39:37.959061 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:37.959082 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:37.959203 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:37.959434 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:37.959437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:37.959625 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:37.959714 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.959719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.960512 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.960515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.961113 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:37.961122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.961577 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.961580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:37.961655 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.962178 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:37.962190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:37.964040 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
W20260927 00:39:37.964341 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:37.958893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51795 (local address 127.0.98.62:39563)
0927 00:39:37.959057 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:37.959062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:37.959166 (+ 104us) server_negotiation.cc:407] Connection header received
0927 00:39:37.959290 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:37.959293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:37.959337 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:37.959402 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:37.959823 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.960404 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.961230 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:37.961473 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:37.961561 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:37.964191 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51795: BlockingRecv error: recv got EOF from 127.0.98.3:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:38.586512 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:38.578580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:50379)
0927 00:39:38.579013 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:38.579030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:38.579180 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:38.579640 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:38.579642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:38.579879 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:38.580009 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.580016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.581499 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.581502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.582161 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.582176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.582886 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.582892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.583004 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.583641 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:38.583655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:38.586265 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":160,"threads_started":1}
W20260927 00:39:38.586623 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:38.578698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50379 (local address 127.0.98.62:39563)
0927 00:39:38.579021 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:38.579027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:38.579140 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:39:38.579364 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:38.579368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:38.579453 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:38.579595 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:38.580149 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.581348 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.582319 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.582719 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.582847 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.586480 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50379: BlockingRecv error: recv got EOF from 127.0.98.1:50379 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:38.783903 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:38.777265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:32969)
0927 00:39:38.777507 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:38.777524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:38.777649 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:38.778041 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:38.778043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:38.778302 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:38.778422 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.778428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.779359 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.779362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.779963 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.779973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.780659 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.780662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.780748 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.781424 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:38.781437 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:38.783677 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
W20260927 00:39:38.784036 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:38.777406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32969 (local address 127.0.98.62:39563)
0927 00:39:38.777621 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:38.777626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:38.777650 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:39:38.777814 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:38.777817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:38.777878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:38.777998 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:38.778572 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.779211 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.780139 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.780488 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.780618 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.783907 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32969: BlockingRecv error: recv got EOF from 127.0.98.2:32969 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:38 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:38.971387 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:38.964995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:51305)
0927 00:39:38.965391 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:38.965414 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:38.965632 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:38.965925 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:38.965928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:38.966174 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:38.966345 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.966353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.967183 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.967187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.967895 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:38.967907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.968420 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.968424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:38.968520 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.969174 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:38.969189 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:38.971184 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
W20260927 00:39:38.971498 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:38.965285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51305 (local address 127.0.98.62:39563)
0927 00:39:38.965451 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:38.965457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:38.965553 (+ 96us) server_negotiation.cc:407] Connection header received
0927 00:39:38.965743 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:38.965746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:38.965801 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:38.965876 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:38.966476 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.967060 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.968016 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:38.968302 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:38.968401 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:38.971378 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51305: BlockingRecv error: recv got EOF from 127.0.98.3:51305 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:39.594851 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:39.587567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:55855)
0927 00:39:39.587990 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:39.588008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:39.588164 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:39.588651 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:39.588654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:39.588885 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:39.589036 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.589051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.589957 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.589961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.590833 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.590853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.591486 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.591490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.591584 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.592306 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:39.592320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:39.594675 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":92,"threads_started":1}
W20260927 00:39:39.595012 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:39.587700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55855 (local address 127.0.98.62:39563)
0927 00:39:39.587974 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:39.587977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:39.588134 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:39:39.588345 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:39.588348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:39.588401 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:39.588499 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:39.589205 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.589784 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.591074 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.591326 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.591419 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.594869 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55855: BlockingRecv error: recv got EOF from 127.0.98.1:55855 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":64,"threads_started":1}
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:39.791857 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:39.785153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:46783)
0927 00:39:39.785663 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:39.785682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:39.785943 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:39.786335 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:39.786339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:39.786538 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:39.786635 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.786641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.787436 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.787440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.788027 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.788036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.788547 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.788551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.788632 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.789257 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:39.789276 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:39.791647 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":156,"threads_started":1}
W20260927 00:39:39.792078 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:39.785361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46783 (local address 127.0.98.62:39563)
0927 00:39:39.785651 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:39.785657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:39.785804 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:39:39.786076 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:39.786080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:39.786134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:39.786232 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:39.786797 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.787324 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.788143 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.788428 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.788515 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.791934 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46783: BlockingRecv error: recv got EOF from 127.0.98.2:46783 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:39 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:39.979301 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:39.972815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:41849)
0927 00:39:39.973290 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:39.973309 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:39.973462 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:39.973822 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:39.973826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:39.974031 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:39.974183 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.974192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.975243 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.975246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.975921 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:39.975933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.976607 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.976610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:39.976702 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.977292 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:39.977308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:39.979147 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":122,"threads_started":1}
W20260927 00:39:39.979678 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:39.973025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41849 (local address 127.0.98.62:39563)
0927 00:39:39.973236 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:39.973242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:39.973501 (+ 259us) server_negotiation.cc:407] Connection header received
0927 00:39:39.973575 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:39.973579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:39.973663 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:39.973755 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:39.974366 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.975123 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.976080 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:39.976411 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:39.976556 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:39.979474 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41849: BlockingRecv error: recv got EOF from 127.0.98.3:41849 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:40.602843 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:40.596057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:51483)
0927 00:39:40.596376 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:40.596393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:40.596563 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:40.597060 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:40.597062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:40.597253 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:40.597391 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.597396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.598582 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.598585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.599173 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.599182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.599765 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.599768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.599846 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.600514 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:40.600527 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:40.602687 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
W20260927 00:39:40.603116 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:40.596244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51483 (local address 127.0.98.62:39563)
0927 00:39:40.596526 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:40.596530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:40.596550 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:39:40.596781 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:40.596786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:40.596866 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:40.597002 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:40.597597 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.598416 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.599339 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.599612 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.599742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.602942 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51483: BlockingRecv error: recv got EOF from 127.0.98.1:51483 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:40.799577 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:40.793138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:35711)
0927 00:39:40.793506 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:40.793520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:40.793638 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:40.793980 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:40.793983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:40.794174 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:40.794302 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.794309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.795488 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.795491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.796074 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.796083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.796671 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.796674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.796749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.797279 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:40.797291 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:40.799437 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
W20260927 00:39:40.799747 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:40.793194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35711 (local address 127.0.98.62:39563)
0927 00:39:40.793359 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:40.793364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:40.793616 (+ 252us) server_negotiation.cc:407] Connection header received
0927 00:39:40.793757 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:40.793760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:40.793842 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:40.793942 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:40.794468 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.795277 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.796258 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.796525 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.796645 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.799610 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35711: BlockingRecv error: recv got EOF from 127.0.98.2:35711 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:40 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:40.994077 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:40.980331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:50341)
0927 00:39:40.980781 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:40.980800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:40.981102 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:40.981548 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:40.981552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:40.981878 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:40.982044 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.982053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.983138 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.983144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.983840 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:40.983852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.984457 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.984462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:40.984554 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.990977 (+ 6423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:40.990998 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:40.993900 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":100,"threads_started":1}
W20260927 00:39:40.994320 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:40.980603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50341 (local address 127.0.98.62:39563)
0927 00:39:40.980764 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:40.980768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:40.981016 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:39:40.981257 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:40.981260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:40.981320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:40.981419 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:40.982237 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.982996 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.983983 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:40.984296 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:40.984412 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:40.994176 (+ 9764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50341: BlockingRecv error: recv got EOF from 127.0.98.3:50341 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:41.610767 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:41.603962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:59821)
0927 00:39:41.604328 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:41.604343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:41.604501 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:41.604934 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:41.604938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:41.605210 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:41.605389 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.605398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.606206 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.606209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.606891 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.606901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.607469 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.607472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.607544 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:41.608317 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:41.608330 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:41.610593 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260927 00:39:41.610893 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:41.604038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59821 (local address 127.0.98.62:39563)
0927 00:39:41.604336 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:41.604341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:41.604448 (+ 107us) server_negotiation.cc:407] Connection header received
0927 00:39:41.604682 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:41.604685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:41.604749 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:41.604870 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:41.605515 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.606069 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.607082 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.607337 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.607438 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:41.610774 (+ 3336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59821: BlockingRecv error: recv got EOF from 127.0.98.1:59821 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:41.805964 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:41.800471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:55399)
0927 00:39:41.800828 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:41.800839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:41.800943 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:41.801178 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:41.801181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:41.801372 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:41.801459 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.801465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.802199 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.802202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.802753 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.802760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.803185 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.803188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.803263 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:41.803772 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:41.803784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:41.805796 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
W20260927 00:39:41.806231 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:41.800574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55399 (local address 127.0.98.62:39563)
0927 00:39:41.800717 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:41.800722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:41.800906 (+ 184us) server_negotiation.cc:407] Connection header received
0927 00:39:41.801027 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:41.801036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:41.801080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:41.801142 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:41.801569 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.802099 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.802861 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.803079 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.803167 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:41.806031 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55399: BlockingRecv error: recv got EOF from 127.0.98.2:55399 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:41 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:42.001150 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:41.995178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:53101)
0927 00:39:41.995450 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:41.995465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:41.995571 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:41.995807 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:41.995809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:41.995989 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:41.996119 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.996125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.997014 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.997017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.997614 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:41.997625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.998119 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.998122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:41.998188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:41.998804 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:41.998816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:42.000968 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260927 00:39:42.001238 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:41.995273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53101 (local address 127.0.98.62:39563)
0927 00:39:41.995520 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:41.995524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:41.995543 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:39:41.995654 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:41.995657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:41.995705 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:41.995783 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:41.996268 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.996877 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.997747 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:41.997990 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:41.998087 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:42.001143 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53101: BlockingRecv error: recv got EOF from 127.0.98.3:53101 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 00:39:42 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:42 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:42.618011 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:42.611907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:43567)
0927 00:39:42.612240 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:42.612257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:42.612355 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:42.612563 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:42.612565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:42.612756 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:42.612857 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:42.612862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.613880 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.613882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:42.614687 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:42.614698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.615213 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.615215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:42.615276 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:42.615878 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:42.615890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:42.617851 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260927 00:39:42.618134 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:42.612060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43567 (local address 127.0.98.62:39563)
0927 00:39:42.612356 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:42.612360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:42.612375 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:42.612417 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:42.612419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:42.612458 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:42.612539 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:42.612993 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.613748 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.614817 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.615066 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.615200 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:42.618012 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43567: BlockingRecv error: recv got EOF from 127.0.98.1:43567 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":134,"threads_started":1}
Sep 27 00:39:42 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:42 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:42.813165 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:42.806849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:44427)
0927 00:39:42.807105 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:42.807120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:42.807222 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:42.807462 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:42.807465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:42.807664 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:42.807762 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:42.807769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.808588 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.808592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:42.809214 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:42.809224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.809654 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.809657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:42.809743 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:42.810448 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:42.810461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:42.812915 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
W20260927 00:39:42.813232 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:42.807024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44427 (local address 127.0.98.62:39563)
0927 00:39:42.807149 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:42.807153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:42.807172 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:39:42.807314 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:42.807318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:42.807362 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:42.807423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:42.807887 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.808484 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.809325 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:42.809551 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:42.809638 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:42.813084 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44427: BlockingRecv error: recv got EOF from 127.0.98.2:44427 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:43.007902 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:43.002096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:33267)
0927 00:39:43.002406 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:43.002421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:43.002537 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:43.002774 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:43.002777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:43.002972 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:43.003064 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.003069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.003791 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.003794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.004392 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.004400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.004811 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.004814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.004886 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.005514 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:43.005525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:43.007683 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
W20260927 00:39:43.007953 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:43.002275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33267 (local address 127.0.98.62:39563)
0927 00:39:43.002409 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:43.002414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:43.002503 (+ 89us) server_negotiation.cc:407] Connection header received
0927 00:39:43.002624 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:43.002627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:43.002678 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:43.002741 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:43.003179 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.003692 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.004492 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.004711 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.004796 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.007852 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33267: BlockingRecv error: recv got EOF from 127.0.98.3:33267 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:43.625583 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:43.618868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:43227)
0927 00:39:43.619241 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:43.619261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:43.619412 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:43.619748 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:43.619750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:43.619984 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:43.620259 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.620271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.621194 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.621197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.621843 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.621853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.622402 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.622406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.622492 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.623230 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:43.623246 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:43.625419 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
W20260927 00:39:43.625849 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:43.618981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43227 (local address 127.0.98.62:39563)
0927 00:39:43.619312 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:43.619316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:43.619337 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:39:43.619547 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:43.619549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:43.619604 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:43.619701 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:43.620496 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.621060 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.621990 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.622235 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.622331 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.625665 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43227: BlockingRecv error: recv got EOF from 127.0.98.1:43227 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:43 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:43.820436 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:43.814103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:35225)
0927 00:39:43.814445 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:43.814458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:43.814554 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:43.814889 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:43.814891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:43.815075 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:43.815186 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.815191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.816112 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.816115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.816979 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:43.816989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.817717 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.817721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:43.817808 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.818423 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:43.818437 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:43.820297 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
W20260927 00:39:43.820801 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:43.814310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35225 (local address 127.0.98.62:39563)
0927 00:39:43.814512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:43.814515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:43.814537 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:39:43.814658 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:43.814661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:43.814718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:43.814829 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:43.815347 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.815987 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.817157 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:43.817577 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:43.817697 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:43.820644 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35225: BlockingRecv error: recv got EOF from 127.0.98.2:35225 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:44.015086 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:44.008990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:33665)
0927 00:39:44.009345 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:44.009362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:44.009488 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:44.009958 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:44.009961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:44.010140 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:44.010248 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.010254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.011207 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.011213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.011841 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.011850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.012464 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.012467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.012531 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.013128 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:44.013140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:44.014954 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
W20260927 00:39:44.015481 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:44.009121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33665 (local address 127.0.98.62:39563)
0927 00:39:44.009330 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:44.009335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:44.009448 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:39:44.009579 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:44.009582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:44.009691 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:44.009781 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:44.010426 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.011053 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.012018 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.012329 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.012448 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.015310 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33665: BlockingRecv error: recv got EOF from 127.0.98.3:33665 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:44.632880 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:44.626554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:44695)
0927 00:39:44.626959 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:44.626975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:44.627104 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:44.627433 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:44.627436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:44.627629 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:44.627743 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.627748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.628720 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.628723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.629443 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.629455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.630009 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.630012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.630084 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.630707 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:44.630719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:44.632734 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
W20260927 00:39:44.633185 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:44.626654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44695 (local address 127.0.98.62:39563)
0927 00:39:44.626965 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:44.626970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:44.627057 (+ 87us) server_negotiation.cc:407] Connection header received
0927 00:39:44.627193 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:44.627195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:44.627251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:44.627352 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:44.627885 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.628581 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.629591 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.629860 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.629959 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.632994 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44695: BlockingRecv error: recv got EOF from 127.0.98.1:44695 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:44 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:44.828398 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:44.821467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:42209)
0927 00:39:44.821895 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:44.821917 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:44.822069 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:44.822450 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:44.822453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:44.822645 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:44.822754 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.822760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.823704 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.823707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.824441 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:44.824453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.825085 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.825089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:44.825165 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.825816 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:44.825834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:44.828258 (+ 2424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
W20260927 00:39:44.828606 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:44.821662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42209 (local address 127.0.98.62:39563)
0927 00:39:44.821872 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:44.821876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:44.822071 (+ 195us) server_negotiation.cc:407] Connection header received
0927 00:39:44.822180 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:44.822182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:44.822256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:44.822339 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:44.822935 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.823537 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.824679 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:44.824933 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:44.825032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:44.828431 (+ 3399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42209: BlockingRecv error: recv got EOF from 127.0.98.2:42209 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:45.022545 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:45.016078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:57337)
0927 00:39:45.016493 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:45.016508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:45.016650 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:45.016964 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:45.016967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:45.017428 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:45.017574 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.017584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.018468 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.018472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.019090 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.019100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.019555 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.019559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.019634 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.020244 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:45.020260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:45.022378 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":115,"threads_started":1}
W20260927 00:39:45.022691 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:45.016240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57337 (local address 127.0.98.62:39563)
0927 00:39:45.016415 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:45.016420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:45.016592 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:39:45.016746 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:45.016749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:45.016844 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:45.016919 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:45.017790 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.018346 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.019216 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.019442 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.019541 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.022544 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57337: BlockingRecv error: recv got EOF from 127.0.98.3:57337 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:45.640329 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:45.633927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:40301)
0927 00:39:45.634328 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:45.634345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:45.634463 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:45.634778 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:45.634783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:45.635004 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:45.635110 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.635117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.635938 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.635942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.636540 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.636549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.637015 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.637019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.637099 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.637667 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:45.637684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:45.640111 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
W20260927 00:39:45.640669 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:45.634051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40301 (local address 127.0.98.62:39563)
0927 00:39:45.634458 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:45.634463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:45.634488 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:39:45.634547 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:45.634550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:45.634601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:45.634719 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:45.635251 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.635820 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.636664 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.636892 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.637000 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.640485 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40301: BlockingRecv error: recv got EOF from 127.0.98.1:40301 (error 108)
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":101,"threads_started":1}
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:45 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:45.834914 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:45.829263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:53637)
0927 00:39:45.829557 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:45.829573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:45.829678 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:45.829911 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:45.829914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:45.830104 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:45.830193 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.830199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.831046 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.831050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.831652 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:45.831661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.832121 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.832124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:45.832232 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.832723 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:45.832736 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:45.834761 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
W20260927 00:39:45.835147 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:45.829476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53637 (local address 127.0.98.62:39563)
0927 00:39:45.829609 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:45.829614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:45.829628 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:39:45.829763 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:45.829766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:45.829812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:45.829873 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:45.830298 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.830947 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.831763 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:45.832015 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:45.832102 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:45.834933 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53637: BlockingRecv error: recv got EOF from 127.0.98.2:53637 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:46.035557 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:46.023580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:39829)
0927 00:39:46.023854 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:46.023866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:46.023971 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:46.024352 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:46.024355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:46.024546 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:46.024649 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.024655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.025555 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.025558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.026180 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.026189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.026662 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.026665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.026757 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.032945 (+ 6188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:46.032961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:46.035382 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
W20260927 00:39:46.035763 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:46.023699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39829 (local address 127.0.98.62:39563)
0927 00:39:46.023848 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:46.023852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:46.023933 (+ 81us) server_negotiation.cc:407] Connection header received
0927 00:39:46.024125 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:46.024127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:46.024183 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:46.024277 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:46.024792 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.025417 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.026293 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.026507 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.026591 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.035613 (+ 9022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39829: BlockingRecv error: recv got EOF from 127.0.98.3:39829 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:46.646977 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:46.641307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:47807)
0927 00:39:46.641568 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:46.641588 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:46.641676 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:46.641921 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:46.641923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:46.642131 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:46.642234 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.642239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.643122 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.643125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.643708 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.643717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.644286 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.644290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.644354 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.644833 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:46.644843 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:46.646816 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260927 00:39:46.647228 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:46.641357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47807 (local address 127.0.98.62:39563)
0927 00:39:46.641678 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:46.641681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:46.641694 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:46.641731 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:46.641733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:46.641773 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:46.641848 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:46.642378 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.642940 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.643870 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.644103 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.644244 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.646984 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47807: BlockingRecv error: recv got EOF from 127.0.98.1:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":70,"threads_started":1}
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:46 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:46.841660 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:46.835853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:37795)
0927 00:39:46.836130 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:46.836144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:46.836295 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:46.836582 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:46.836585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:46.836807 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:46.836901 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.836907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.837771 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.837774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.838401 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:46.838412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.838934 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.838937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:46.839016 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.839585 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:46.839597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:46.841518 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
W20260927 00:39:46.841897 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:46.835913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37795 (local address 127.0.98.62:39563)
0927 00:39:46.836068 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:46.836073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:46.836244 (+ 171us) server_negotiation.cc:407] Connection header received
0927 00:39:46.836391 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:46.836394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:46.836443 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:46.836517 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:46.837046 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.837646 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.838533 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:46.838757 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:46.838893 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:46.841708 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37795: BlockingRecv error: recv got EOF from 127.0.98.2:37795 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:47.042770 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:47.036512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:52115)
0927 00:39:47.036865 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:47.036878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:47.036975 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:47.037290 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:47.037293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:47.037472 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:47.037569 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.037575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.038358 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.038361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.038984 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.038994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.039518 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.039522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.039613 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.040147 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:47.040161 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:47.042606 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":74,"threads_started":1}
W20260927 00:39:47.042894 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:47.036579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52115 (local address 127.0.98.62:39563)
0927 00:39:47.036695 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:47.036700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:47.036972 (+ 272us) server_negotiation.cc:407] Connection header received
0927 00:39:47.037044 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:47.037047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:47.037103 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:47.037274 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:47.037692 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.038223 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.039115 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.039401 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.039489 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.042774 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52115: BlockingRecv error: recv got EOF from 127.0.98.3:52115 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:47.653770 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:47.648097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:48767)
0927 00:39:47.648346 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:47.648363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:47.648474 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:47.648701 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:47.648703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:47.648880 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:47.649017 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.649023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.649844 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.649847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.650450 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.650458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.650949 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.650952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.651022 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.651560 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:47.651573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:47.653582 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":123,"threads_started":1}
W20260927 00:39:47.653862 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:47.648147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48767 (local address 127.0.98.62:39563)
0927 00:39:47.648475 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:47.648478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:47.648493 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:39:47.648546 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:47.648548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:47.648588 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:47.648674 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:47.649131 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.649733 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.650566 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.650813 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.650894 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.653749 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48767: BlockingRecv error: recv got EOF from 127.0.98.1:48767 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":92,"threads_started":1}
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:39:47 dist-test-slave-xltq krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:39:47.848354 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:47.842690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:36693)
0927 00:39:47.842953 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:47.842966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:47.843068 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:47.843342 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:47.843345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:47.843516 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:47.843610 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.843615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.844468 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.844471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.845057 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:47.845065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.845559 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.845562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:47.845630 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.846159 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:47.846170 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:47.848182 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39563: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
W20260927 00:39:47.848526 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:47.842857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36693 (local address 127.0.98.62:39563)
0927 00:39:47.842978 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:47.842982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:47.843090 (+ 108us) server_negotiation.cc:407] Connection header received
0927 00:39:47.843149 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:47.843151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:47.843197 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:47.843278 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:47.843724 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.844358 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.845175 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:47.845392 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:47.845522 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:47.848389 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36693: BlockingRecv error: recv got EOF from 127.0.98.2:36693 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:39:47.909299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9141
I20260927 00:39:47.914391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9293
I20260927 00:39:47.919202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9425
I20260927 00:39:47.923661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9071
2026-09-27T00:39:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17927 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:47Z Disabled control of system clock
I20260927 00:39:47.952688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:39:48.065541 9621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:48.065883 9621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:48.065922 9621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:48.070158 9621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:48.070225 9621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:48.070240 9621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:48.070254 9621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:48.070268 9621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:48.075266 9621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:48.076447 9621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:48.077910 9621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:48.087185 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.087301 9621 server_base.cc:1034] running on GCE node
W20260927 00:39:48.087344 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:48.087191 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.088083 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:48.089439 9621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:48.090628 9621 hybrid_clock.cc:648] HybridClock initialized: now 1790469588090604 us; error 42 us; skew 500 ppm
I20260927 00:39:48.092782 9621 webserver.cc:460] Webserver started at http://127.0.98.62:37877/ using document root <none> and password file <none>
I20260927 00:39:48.093372 9621 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:48.093442 9621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:48.093652 9621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:48.095788 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "6e2ef061448b4f909f0864923acdb33e"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "0a13639968f973b126605ee7effabcca"
server_key_iv: "561fbb6106c4cf76d3510c858ee8347d"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.096378 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e2ef061448b4f909f0864923acdb33e"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "0a13639968f973b126605ee7effabcca"
server_key_iv: "561fbb6106c4cf76d3510c858ee8347d"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.100080 9621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:48.102913 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.104416 9621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:48.104537 9621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "6e2ef061448b4f909f0864923acdb33e"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "0a13639968f973b126605ee7effabcca"
server_key_iv: "561fbb6106c4cf76d3510c858ee8347d"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.104647 9621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:48.122236 9621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:48.122982 9621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:48.123162 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:48.130093 9621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42223
I20260927 00:39:48.130165 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42223 every 8 connection(s)
I20260927 00:39:48.131171 9621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:48.134626 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:48.138943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9621
I20260927 00:39:48.139096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:48.139441 392 external_mini_cluster.cc:1428] Setting key 203949b342d3599b0c4a74cdc5d096e0
I20260927 00:39:48.141520 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e: Bootstrap starting.
I20260927 00:39:48.144311 9688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:48.145200 9688 log.cc:824] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:48.147528 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e: No bootstrap required, opened a new log
I20260927 00:39:48.150174 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.141099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.62:42223)
0927 00:39:48.141771 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:48.141788 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:48.141827 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:39:48.142520 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:48.142538 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:48.142972 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:48.143288 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:48.143680 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.144893 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.146121 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.146668 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.146806 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.147782 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:48.147789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:48.147807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:48.148380 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:48.148457 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:48.148487 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:48.148585 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:48.148823 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:39:48.149138 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260927 00:39:48.152129 9688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } }
I20260927 00:39:48.152424 9688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:48.152509 9688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2ef061448b4f909f0864923acdb33e, State: Initialized, Role: FOLLOWER
I20260927 00:39:48.153210 9688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } }
I20260927 00:39:48.153365 9688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:48.153436 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:48.153561 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:48.154621 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } }
I20260927 00:39:48.154966 9688 leader_election.cc:302] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e2ef061448b4f909f0864923acdb33e; no voters:
I20260927 00:39:48.155324 9688 leader_election.cc:288] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:48.155541 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:48.155802 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [term 1 LEADER]: Becoming Leader. State: Replica: 6e2ef061448b4f909f0864923acdb33e, State: Running, Role: LEADER
I20260927 00:39:48.156152 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } }
I20260927 00:39:48.156647 9688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:48.157685 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e2ef061448b4f909f0864923acdb33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } } }
I20260927 00:39:48.157812 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:48.158143 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e2ef061448b4f909f0864923acdb33e. Latest consensus state: current_term: 1 leader_uuid: "6e2ef061448b4f909f0864923acdb33e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e2ef061448b4f909f0864923acdb33e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42223 } } }
I20260927 00:39:48.158257 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:48.158458 9702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:48.161492 9702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:48.167178 9702 catalog_manager.cc:1347] Generated new cluster ID: 728a0161c71740e19df093fa154c04e9
I20260927 00:39:48.167272 9702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:48.198212 9702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:48.199510 9702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:48.211694 9702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e2ef061448b4f909f0864923acdb33e: Generated new TSK 0
I20260927 00:39:48.212714 9702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:48.221087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--builtin_ntp_servers=127.0.98.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:39:48.339484 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:48.339785 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:48.339838 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:48.344058 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:48.344133 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:48.344234 9712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:48.349315 9712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:48.350468 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:48.351877 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:48.362219 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.362249 9712 server_base.cc:1034] running on GCE node
W20260927 00:39:48.362207 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:48.362207 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.362933 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:48.363644 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:48.364843 9712 hybrid_clock.cc:648] HybridClock initialized: now 1790469588364809 us; error 49 us; skew 500 ppm
I20260927 00:39:48.367183 9712 webserver.cc:460] Webserver started at http://127.0.98.1:44631/ using document root <none> and password file <none>
I20260927 00:39:48.367859 9712 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:48.367938 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:48.368140 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:48.370402 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "22ff7d5985c54d19a2b58a43d767a21c"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "02969d389d17a624f3d204b01d1b6c54"
server_key_iv: "3b558cd36555cac7577ac57d06c63ba0"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.371004 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22ff7d5985c54d19a2b58a43d767a21c"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "02969d389d17a624f3d204b01d1b6c54"
server_key_iv: "3b558cd36555cac7577ac57d06c63ba0"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.375279 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:48.378361 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.379860 9712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:39:48.379983 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "22ff7d5985c54d19a2b58a43d767a21c"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "02969d389d17a624f3d204b01d1b6c54"
server_key_iv: "3b558cd36555cac7577ac57d06c63ba0"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.380105 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:48.404568 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:48.405334 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:48.405527 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:48.406168 9712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:48.407238 9712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:48.407301 9712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.407364 9712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:48.407394 9712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.417371 9712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46857
I20260927 00:39:48.417439 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46857 every 8 connection(s)
I20260927 00:39:48.418430 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:48.428642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9712
I20260927 00:39:48.428767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:48.429147 392 external_mini_cluster.cc:1428] Setting key 28bcb712b73d8c0ed9f82e9a3731467e
I20260927 00:39:48.429390 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.420553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49037 (local address 127.0.98.62:42223)
0927 00:39:48.420714 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:48.420719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:48.421947 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 00:39:48.423446 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:48.423449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:48.423523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:48.423610 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:48.424939 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.425503 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.426894 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.427126 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.427223 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.428713 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:48.428717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:48.428720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:48.428738 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:48.428766 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:48.428777 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:48.428873 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:48.429056 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 00:39:48.429147 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:39:48.430460 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.420985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42223 (local address 127.0.98.1:49037)
0927 00:39:48.421755 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:48.421804 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:48.423174 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:48.423898 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:48.423907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:48.424014 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:48.424698 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.424715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.425669 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.425672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.426693 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.426704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.427411 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.427416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.427602 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.428320 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:48.428322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:48.428369 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:48.428524 (+ 155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:48.428876 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:48.428887 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:48.428893 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:48.429038 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:39:48.429339 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260927 00:39:48.431267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--builtin_ntp_servers=127.0.98.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:39:48.431850 9840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42223
I20260927 00:39:48.432140 9840 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:48.432806 9840 heartbeater.cc:511] Master 127.0.98.62:42223 requested a full tablet report, sending...
I20260927 00:39:48.434578 9652 ts_manager.cc:194] Registered new tserver with Master: 22ff7d5985c54d19a2b58a43d767a21c (127.0.98.1:46857)
I20260927 00:39:48.435748 9652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49037
W20260927 00:39:48.543274 9843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:48.543577 9843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:48.543638 9843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:48.547861 9843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:48.547960 9843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:48.548043 9843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:48.553069 9843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:48.554308 9843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:48.555840 9843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:48.566614 9843 server_base.cc:1034] running on GCE node
W20260927 00:39:48.566669 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:48.566582 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:48.566648 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.567457 9843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:48.568244 9843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:48.569545 9843 hybrid_clock.cc:648] HybridClock initialized: now 1790469588569522 us; error 40 us; skew 500 ppm
I20260927 00:39:48.571712 9843 webserver.cc:460] Webserver started at http://127.0.98.2:46065/ using document root <none> and password file <none>
I20260927 00:39:48.572372 9843 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:48.572453 9843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:48.572661 9843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:48.574872 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f7a4f43961a4f93934ea4e128755547"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "35189090edb6664bf08587573be7e72a"
server_key_iv: "e4efd9d456ddb8eab9abe934a1682f9b"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.575456 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f7a4f43961a4f93934ea4e128755547"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "35189090edb6664bf08587573be7e72a"
server_key_iv: "e4efd9d456ddb8eab9abe934a1682f9b"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.579662 9843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:39:48.582724 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.584473 9843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:48.584613 9843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "8f7a4f43961a4f93934ea4e128755547"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "35189090edb6664bf08587573be7e72a"
server_key_iv: "e4efd9d456ddb8eab9abe934a1682f9b"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.584750 9843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:48.604038 9843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:48.604853 9843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:48.605043 9843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:48.605657 9843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:48.606719 9843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:48.606768 9843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.606824 9843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:48.606850 9843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.616004 9843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44651
I20260927 00:39:48.616039 9970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44651 every 8 connection(s)
I20260927 00:39:48.616991 9843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:48.618001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9843
I20260927 00:39:48.618094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:48.618366 392 external_mini_cluster.cc:1428] Setting key 1f32babac79c4c61daafad7d11cdcd00
I20260927 00:39:48.620555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--builtin_ntp_servers=127.0.98.20:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:39:48.626844 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.619027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43845 (local address 127.0.98.62:42223)
0927 00:39:48.619165 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:48.619168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:48.620612 (+ 1444us) server_negotiation.cc:407] Connection header received
0927 00:39:48.622041 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:48.622045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:48.622102 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:48.622176 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:48.623143 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.623740 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.624941 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.625176 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.625265 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.626371 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:48.626375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:48.626378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:48.626392 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:48.626412 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:48.626420 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:48.626477 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:48.626627 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 00:39:48.626691 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:39:48.627811 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.619557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42223 (local address 127.0.98.2:43845)
0927 00:39:48.620428 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:48.620487 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:48.621810 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:48.622370 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:48.622378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:48.622474 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:48.622999 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.623013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.623868 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.623871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.624821 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.624832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.625326 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.625331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.625512 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.626100 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:48.626101 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:48.626145 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:48.626219 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:48.626488 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:48.626498 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:48.626504 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:48.626624 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:39:48.626932 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":173,"threads_started":1}
I20260927 00:39:48.629001 9971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42223
I20260927 00:39:48.629305 9971 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:48.630000 9971 heartbeater.cc:511] Master 127.0.98.62:42223 requested a full tablet report, sending...
I20260927 00:39:48.631275 9652 ts_manager.cc:194] Registered new tserver with Master: 8f7a4f43961a4f93934ea4e128755547 (127.0.98.2:44651)
I20260927 00:39:48.631956 9652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43845
W20260927 00:39:48.737891 9974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:48.738188 9974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:48.738227 9974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:48.742413 9974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:48.742486 9974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:48.742547 9974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:48.747646 9974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:48.748811 9974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:48.750226 9974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:48.760815 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:48.760767 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.760882 9974 server_base.cc:1034] running on GCE node
W20260927 00:39:48.760815 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:48.761593 9974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:48.762301 9974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:48.763491 9974 hybrid_clock.cc:648] HybridClock initialized: now 1790469588763456 us; error 49 us; skew 500 ppm
I20260927 00:39:48.765630 9974 webserver.cc:460] Webserver started at http://127.0.98.3:34405/ using document root <none> and password file <none>
I20260927 00:39:48.766249 9974 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:48.766316 9974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:48.766505 9974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:48.768961 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "3644ffc71cb148b5bf67f90eec1b710b"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "5eedaa66920539c2cba975e3ef7d76eb"
server_key_iv: "f385b55c614ca0bc8a925277d4aa6f09"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.769590 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3644ffc71cb148b5bf67f90eec1b710b"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "5eedaa66920539c2cba975e3ef7d76eb"
server_key_iv: "f385b55c614ca0bc8a925277d4aa6f09"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.773885 9974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:39:48.776958 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.778502 9974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:39:48.778622 9974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "3644ffc71cb148b5bf67f90eec1b710b"
format_stamp: "Formatted at 2026-09-27 00:39:48 on dist-test-slave-xltq"
server_key: "5eedaa66920539c2cba975e3ef7d76eb"
server_key_iv: "f385b55c614ca0bc8a925277d4aa6f09"
server_key_version: "encryptionkey@0"
I20260927 00:39:48.778755 9974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:48.804663 9974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:48.805413 9974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:48.805590 9974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:48.806190 9974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:48.807260 9974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:48.807346 9974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.807401 9974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:48.807417 9974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:48.816056 9974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45623
I20260927 00:39:48.816105 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45623 every 8 connection(s)
I20260927 00:39:48.817071 9974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:48.826982 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.819076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58979 (local address 127.0.98.62:42223)
0927 00:39:48.819229 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:48.819233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:48.820517 (+ 1284us) server_negotiation.cc:407] Connection header received
0927 00:39:48.821981 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:48.821984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:48.822034 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:48.822119 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:48.823125 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.823669 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.824957 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.825182 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.825288 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.826437 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:48.826440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:48.826443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:48.826457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:48.826479 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:48.826489 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:48.826567 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:48.826756 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:39:48.826816 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:39:48.827207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 9974
I20260927 00:39:48.827310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:48.827593 392 external_mini_cluster.cc:1428] Setting key 74c7804cb82f13e8e1835fc9c5575cc1
I20260927 00:39:48.827916 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:48.819523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42223 (local address 127.0.98.3:58979)
0927 00:39:48.820267 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:48.820310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:48.821681 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:48.822286 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:48.822293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:48.822457 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:48.822981 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.822994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.823807 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.823810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.824822 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:48.824832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:48.825340 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:48.825411 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:48.825564 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:48.826150 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:48.826152 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:48.826194 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:48.826260 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:48.826579 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:48.826585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:48.826589 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:39:48.826688 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:39:48.826940 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260927 00:39:48.829154 10102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42223
I20260927 00:39:48.829488 10102 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:48.830013 10102 heartbeater.cc:511] Master 127.0.98.62:42223 requested a full tablet report, sending...
I20260927 00:39:48.831061 9652 ts_manager.cc:194] Registered new tserver with Master: 3644ffc71cb148b5bf67f90eec1b710b (127.0.98.3:45623)
I20260927 00:39:48.831712 9652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58979
I20260927 00:39:48.841732 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:39:48.844758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9712
W20260927 00:39:48.844781 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:48.843613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.0.98.62:42223)
0927 00:39:48.843803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:48.843808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:48.843960 (+ 152us) server_negotiation.cc:407] Connection header received
0927 00:39:48.844104 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:48.844108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:48.844155 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:48.844273 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:48.844624 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37108: BlockingRecv error: recv got EOF from 127.0.0.1:37108 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:39:48.849670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9843
I20260927 00:39:48.854414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9974
I20260927 00:39:48.859558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 9621
2026-09-27T00:39:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (934 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:48Z Disabled control of system clock
I20260927 00:39:48.888141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44161
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:39:49.000458 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:49.000772 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:49.000826 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:49.005195 10113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:49.005280 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:49.005302 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:49.005323 10113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:49.005342 10113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:49.010504 10113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:49.011703 10113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:49.013154 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:49.022068 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.022045 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.022087 10113 server_base.cc:1034] running on GCE node
W20260927 00:39:49.022049 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.022907 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:49.024047 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:49.025216 10113 hybrid_clock.cc:648] HybridClock initialized: now 1790469589025195 us; error 49 us; skew 500 ppm
I20260927 00:39:49.027017 10113 webserver.cc:460] Webserver started at http://127.0.98.62:37309/ using document root <none> and password file <none>
I20260927 00:39:49.027583 10113 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:49.027643 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:49.027803 10113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:49.029908 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "76db8b4ebcb747948158261af77b6ca5"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0d3b7922b3ec87149c93f71c404173b6"
server_key_iv: "161f5322d14532db46cc00e0e0146f4f"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.030427 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "76db8b4ebcb747948158261af77b6ca5"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0d3b7922b3ec87149c93f71c404173b6"
server_key_iv: "161f5322d14532db46cc00e0e0146f4f"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.034536 10113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 00:39:49.037225 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.038741 10113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:49.038856 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "76db8b4ebcb747948158261af77b6ca5"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0d3b7922b3ec87149c93f71c404173b6"
server_key_iv: "161f5322d14532db46cc00e0e0146f4f"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.038964 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:49.043678 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:49.044399 10113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:49.044552 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:49.052011 10113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44161
I20260927 00:39:49.052040 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44161 every 8 connection(s)
I20260927 00:39:49.053104 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:49.054138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10113
I20260927 00:39:49.054239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:49.054571 392 external_mini_cluster.cc:1428] Setting key 2711530899c6ad3eb6b9dd366a6b599c
I20260927 00:39:49.056746 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:49.060992 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.055974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.98.62:44161)
0927 00:39:49.057140 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:49.057159 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:49.057200 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:39:49.057901 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:49.057922 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:49.058329 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:49.058667 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:49.059622 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:49.059627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:49.059640 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:49.059652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:49.059710 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:49.059727 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:49.059843 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:49.059941 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 00:39:49.060234 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":575,"threads_started":1}
I20260927 00:39:49.063372 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5: Bootstrap starting.
I20260927 00:39:49.065677 10180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:49.066406 10180 log.cc:824] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:49.068845 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5: No bootstrap required, opened a new log
I20260927 00:39:49.072510 10180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } }
I20260927 00:39:49.072769 10180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:49.072803 10180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76db8b4ebcb747948158261af77b6ca5, State: Initialized, Role: FOLLOWER
I20260927 00:39:49.073197 10180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } }
I20260927 00:39:49.073313 10180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:49.073362 10180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:49.073443 10180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:49.074415 10180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } }
I20260927 00:39:49.074764 10180 leader_election.cc:302] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76db8b4ebcb747948158261af77b6ca5; no voters:
I20260927 00:39:49.075107 10180 leader_election.cc:288] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:49.075256 10185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:49.075515 10185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 76db8b4ebcb747948158261af77b6ca5, State: Running, Role: LEADER
I20260927 00:39:49.075863 10185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } }
I20260927 00:39:49.076452 10180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:49.077690 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76db8b4ebcb747948158261af77b6ca5. Latest consensus state: current_term: 1 leader_uuid: "76db8b4ebcb747948158261af77b6ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } } }
I20260927 00:39:49.077702 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76db8b4ebcb747948158261af77b6ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76db8b4ebcb747948158261af77b6ca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44161 } } }
I20260927 00:39:49.077876 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:49.078157 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:49.078238 10196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:49.081089 10196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:49.087164 10196 catalog_manager.cc:1347] Generated new cluster ID: ae15f6de7df04a72800903cdb687ebf3
I20260927 00:39:49.087260 10196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:49.095683 10196 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:49.096076 10196 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:49.105795 10196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76db8b4ebcb747948158261af77b6ca5: Generated new TSK 0
I20260927 00:39:49.106725 10196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:39:49.109941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--builtin_ntp_servers=127.0.98.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:39:49.223981 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:49.224318 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:49.224380 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:49.228622 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:49.228694 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:49.228776 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:49.233892 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:49.235077 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:49.236582 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:49.246824 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.246824 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.246824 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.247274 10204 server_base.cc:1034] running on GCE node
I20260927 00:39:49.247825 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:49.248646 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:49.249866 10204 hybrid_clock.cc:648] HybridClock initialized: now 1790469589249851 us; error 53 us; skew 500 ppm
I20260927 00:39:49.252110 10204 webserver.cc:460] Webserver started at http://127.0.98.1:37099/ using document root <none> and password file <none>
I20260927 00:39:49.252768 10204 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:49.252848 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:49.253057 10204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:49.255304 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f801cf282eb48a695f77091d2418038"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "5cd60e870c3bddca374490efae590d8c"
server_key_iv: "3907d262c9e55ee49785f8a4986a21e6"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.256012 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f801cf282eb48a695f77091d2418038"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "5cd60e870c3bddca374490efae590d8c"
server_key_iv: "3907d262c9e55ee49785f8a4986a21e6"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.260183 10204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:39:49.263109 10218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.264668 10204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:49.264796 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "4f801cf282eb48a695f77091d2418038"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "5cd60e870c3bddca374490efae590d8c"
server_key_iv: "3907d262c9e55ee49785f8a4986a21e6"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.264910 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:49.269769 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:49.270445 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:49.270639 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:49.271221 10204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:49.272166 10204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:49.272265 10204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.272330 10204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:49.272361 10204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.281250 10204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43655
I20260927 00:39:49.281276 10331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43655 every 8 connection(s)
I20260927 00:39:49.282161 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:49.286303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10204
I20260927 00:39:49.286420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:49.286686 392 external_mini_cluster.cc:1428] Setting key 76fc24ad2611f7e01d6ebac5847327a6
I20260927 00:39:49.288697 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.283908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50767 (local address 127.0.98.62:44161)
0927 00:39:49.284034 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:49.284037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:49.285332 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 00:39:49.286844 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:49.286848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:49.286896 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:49.287006 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:49.288243 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:49.288247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:49.288249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:49.288263 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:49.288294 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:49.288305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:49.288397 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:49.288512 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:39:49.288558 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:39:49.288769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--builtin_ntp_servers=127.0.98.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:39:49.289836 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.284400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44161 (local address 127.0.98.1:50767)
0927 00:39:49.285145 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:49.285193 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:49.286594 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:49.287245 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:49.287255 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:49.287364 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:49.287944 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:49.287946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:49.287991 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:49.288070 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:49.288383 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:49.288389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:49.288393 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:39:49.288539 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:39:49.288924 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":116,"threads_started":1}
I20260927 00:39:49.291208 10332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44161
I20260927 00:39:49.291440 10332 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:49.291872 10332 heartbeater.cc:511] Master 127.0.98.62:44161 requested a full tablet report, sending...
I20260927 00:39:49.293555 10144 ts_manager.cc:194] Registered new tserver with Master: 4f801cf282eb48a695f77091d2418038 (127.0.98.1:43655)
W20260927 00:39:49.401624 10335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:49.401983 10335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:49.402052 10335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:49.406400 10335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:49.406473 10335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:49.406555 10335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:49.411780 10335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:49.412979 10335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:49.414403 10335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:49.424423 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.424487 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.424609 10335 server_base.cc:1034] running on GCE node
W20260927 00:39:49.424423 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.425199 10335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:49.425845 10335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:49.427038 10335 hybrid_clock.cc:648] HybridClock initialized: now 1790469589427019 us; error 37 us; skew 500 ppm
I20260927 00:39:49.429227 10335 webserver.cc:460] Webserver started at http://127.0.98.2:42865/ using document root <none> and password file <none>
I20260927 00:39:49.429800 10335 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:49.429879 10335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:49.430047 10335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:49.432179 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "01c961adad3b4dd59bcf964e24cb44b9"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0fceec2caa3438043211a41dc3b59bb5"
server_key_iv: "d6e36ff2a74d3c1fa2e7206c6fc31701"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.432799 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "01c961adad3b4dd59bcf964e24cb44b9"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0fceec2caa3438043211a41dc3b59bb5"
server_key_iv: "d6e36ff2a74d3c1fa2e7206c6fc31701"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.436834 10335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:39:49.439500 10349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.440936 10335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:49.441052 10335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "01c961adad3b4dd59bcf964e24cb44b9"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "0fceec2caa3438043211a41dc3b59bb5"
server_key_iv: "d6e36ff2a74d3c1fa2e7206c6fc31701"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.441160 10335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:49.446136 10335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:49.446779 10335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:49.446921 10335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:49.447454 10335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:49.448458 10335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:49.448508 10335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.448549 10335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:49.448563 10335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.457324 10335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37625
I20260927 00:39:49.457350 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37625 every 8 connection(s)
I20260927 00:39:49.458318 10335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:49.464781 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.460290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52747 (local address 127.0.98.62:44161)
0927 00:39:49.460437 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:49.460441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:49.461713 (+ 1272us) server_negotiation.cc:407] Connection header received
0927 00:39:49.462979 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:49.462982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:49.463020 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:49.463083 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:49.464352 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:49.464355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:49.464357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:49.464371 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:49.464392 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:49.464401 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:49.464504 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:49.464628 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 00:39:49.464667 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:39:49.465662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10335
I20260927 00:39:49.465766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:49.465834 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.460750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44161 (local address 127.0.98.2:52747)
0927 00:39:49.461539 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:49.461585 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:49.462766 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:49.463251 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:49.463258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:49.463355 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:49.464019 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:49.464020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:49.464070 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:49.464149 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:49.464499 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:49.464506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:49.464511 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:39:49.464663 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:39:49.464889 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260927 00:39:49.466076 392 external_mini_cluster.cc:1428] Setting key 25e4c606801e122e183b8e37e99fb19f
I20260927 00:39:49.467139 10463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44161
I20260927 00:39:49.467393 10463 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:49.467818 10463 heartbeater.cc:511] Master 127.0.98.62:44161 requested a full tablet report, sending...
I20260927 00:39:49.467916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--builtin_ntp_servers=127.0.98.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:39:49.469094 10144 ts_manager.cc:194] Registered new tserver with Master: 01c961adad3b4dd59bcf964e24cb44b9 (127.0.98.2:37625)
W20260927 00:39:49.583739 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:49.584043 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:49.584085 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:49.588321 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:49.588390 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:49.588450 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:49.593446 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:49.594589 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:49.595984 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:49.605880 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.606006 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:49.605876 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:49.606029 10466 server_base.cc:1034] running on GCE node
I20260927 00:39:49.606609 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:49.607362 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:49.608531 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790469589608508 us; error 36 us; skew 500 ppm
I20260927 00:39:49.610677 10466 webserver.cc:460] Webserver started at http://127.0.98.3:36927/ using document root <none> and password file <none>
I20260927 00:39:49.611289 10466 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:49.611361 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:49.611558 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:49.613845 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "c82405b143c54d85bc06153cf0737ff1"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "b565707fc17a42cb9324f47385fb8bc6"
server_key_iv: "08515de8abed335cb541de5b6adb640d"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.614434 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c82405b143c54d85bc06153cf0737ff1"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "b565707fc17a42cb9324f47385fb8bc6"
server_key_iv: "08515de8abed335cb541de5b6adb640d"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.618643 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:39:49.621341 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.622783 10466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:49.622911 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "c82405b143c54d85bc06153cf0737ff1"
format_stamp: "Formatted at 2026-09-27 00:39:49 on dist-test-slave-xltq"
server_key: "b565707fc17a42cb9324f47385fb8bc6"
server_key_iv: "08515de8abed335cb541de5b6adb640d"
server_key_version: "encryptionkey@0"
I20260927 00:39:49.623032 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:49.627919 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:49.628638 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:49.628808 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:49.629374 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:49.630311 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:49.630364 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.630419 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:49.630443 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:49.639343 10466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38991
I20260927 00:39:49.639416 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38991 every 8 connection(s)
I20260927 00:39:49.640372 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:49.645849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10466
I20260927 00:39:49.645957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:49.646242 392 external_mini_cluster.cc:1428] Setting key 9f4f5a55eb5068e1b90ede59afd1a1ec
I20260927 00:39:49.646812 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.642226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39651 (local address 127.0.98.62:44161)
0927 00:39:49.642354 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:49.642357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:49.643568 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 00:39:49.644989 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:49.644992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:49.645038 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:49.645113 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:49.646341 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:49.646345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:49.646348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:39:49.646362 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:49.646386 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:39:49.646396 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:49.646480 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:49.646620 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 00:39:49.646676 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:39:49.647922 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:49.642651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44161 (local address 127.0.98.3:39651)
0927 00:39:49.643423 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:49.643468 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:49.644767 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:49.645323 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:49.645332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:49.645442 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:49.646033 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:39:49.646035 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:49.646093 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:39:49.646182 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:49.646471 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:49.646480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:49.646486 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:39:49.646662 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:39:49.646933 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260927 00:39:49.649178 10594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44161
I20260927 00:39:49.649467 10594 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:49.649995 10594 heartbeater.cc:511] Master 127.0.98.62:44161 requested a full tablet report, sending...
I20260927 00:39:49.651269 10144 ts_manager.cc:194] Registered new tserver with Master: c82405b143c54d85bc06153cf0737ff1 (127.0.98.3:38991)
I20260927 00:39:49.661244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:39:49.664265 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:39:49.663177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.98.62:44161)
0927 00:39:49.663351 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:49.663356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:49.663448 (+ 92us) server_negotiation.cc:407] Connection header received
0927 00:39:49.663682 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:49.663686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:49.663726 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:49.663804 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:49.664084 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60594: BlockingRecv error: recv got EOF from 127.0.0.1:60594 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:39:49.664426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10204
I20260927 00:39:49.669166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10335
I20260927 00:39:49.674080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10466
I20260927 00:39:49.678743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10113
2026-09-27T00:39:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (818 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:49 dist-test-slave-xltq krb5kdc[10604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:49 dist-test-slave-xltq krb5kdc[10604](info): set up 2 sockets
Sep 27 00:39:49 dist-test-slave-xltq krb5kdc[10604](info): commencing operation
krb5kdc: starting...
W20260927 00:39:50.637055 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:50 dist-test-slave-xltq krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:39:50.761077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:50.880389 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:50.880688 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:50.880731 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:50.884912 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:39:50.884990 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:50.885010 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:50.885026 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:39:50.885040 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:39:50.890698 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:50.891876 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:50.893128 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:50.901922 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:50.901975 10617 server_base.cc:1034] running on GCE node
W20260927 00:39:50.901937 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:50.901922 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:50.902660 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:50.903841 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:50.905030 10617 hybrid_clock.cc:648] HybridClock initialized: now 1790469590905003 us; error 40 us; skew 500 ppm
Sep 27 00:39:50 dist-test-slave-xltq krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:50.907845 10617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:39:50.908880 10617 webserver.cc:460] Webserver started at http://127.0.98.62:40891/ using document root <none> and password file <none>
I20260927 00:39:50.909461 10617 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:50.909533 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:50.909730 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:50.911782 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "87285e1acf96486aa31209535a7f6604"
format_stamp: "Formatted at 2026-09-27 00:39:50 on dist-test-slave-xltq"
server_key: "6749dc012327d2f6a1af019f536da2c8"
server_key_iv: "17dec371f3f08d8b4133cb7dc4c95b94"
server_key_version: "encryptionkey@0"
I20260927 00:39:50.912392 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "87285e1acf96486aa31209535a7f6604"
format_stamp: "Formatted at 2026-09-27 00:39:50 on dist-test-slave-xltq"
server_key: "6749dc012327d2f6a1af019f536da2c8"
server_key_iv: "17dec371f3f08d8b4133cb7dc4c95b94"
server_key_version: "encryptionkey@0"
I20260927 00:39:50.916028 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:50.918902 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:50.920378 10617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:39:50.920514 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "87285e1acf96486aa31209535a7f6604"
format_stamp: "Formatted at 2026-09-27 00:39:50 on dist-test-slave-xltq"
server_key: "6749dc012327d2f6a1af019f536da2c8"
server_key_iv: "17dec371f3f08d8b4133cb7dc4c95b94"
server_key_version: "encryptionkey@0"
I20260927 00:39:50.920629 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:50.935582 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:50.938934 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:50.939126 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:50.946156 10617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46259
I20260927 00:39:50.946496 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46259 every 8 connection(s)
I20260927 00:39:50.947160 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:39:50.948079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10617
I20260927 00:39:50.948230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:39:50.948540 392 external_mini_cluster.cc:1428] Setting key 4d63f62b090df8dc8b852bb5794788e2
I20260927 00:39:50.950556 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:50.956991 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: Bootstrap starting.
Sep 27 00:39:50 dist-test-slave-xltq krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:50.959451 10685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:50.960232 10685 log.cc:824] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:50.962563 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: No bootstrap required, opened a new log
I20260927 00:39:50.963236 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:50.949901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.98.62:46259)
0927 00:39:50.950291 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:50.950306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:50.950350 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:39:50.951024 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:50.951042 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:50.951380 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:50.951666 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:50.952494 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:50.953530 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:50.954649 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:50.955109 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:50.955226 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:50.958018 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:50.958025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:50.958043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:50.958081 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:50.959953 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:50.960464 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:50.960470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:50.960475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:50.960595 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:50.960894 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:50.960897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:50.960898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:50.961364 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:50.961515 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:50.962050 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 00:39:50.962408 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260927 00:39:50.966637 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260927 00:39:50.966903 10685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:50.966964 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87285e1acf96486aa31209535a7f6604, State: Initialized, Role: FOLLOWER
I20260927 00:39:50.967371 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260927 00:39:50.967502 10685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:39:50.967566 10685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:39:50.967666 10685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:50.968801 10685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260927 00:39:50.969165 10685 leader_election.cc:302] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87285e1acf96486aa31209535a7f6604; no voters:
I20260927 00:39:50.969521 10685 leader_election.cc:288] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:39:50.969669 10690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:50.969916 10690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [term 1 LEADER]: Becoming Leader. State: Replica: 87285e1acf96486aa31209535a7f6604, State: Running, Role: LEADER
I20260927 00:39:50.970290 10690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260927 00:39:50.970935 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:39:50.972244 10692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87285e1acf96486aa31209535a7f6604. Latest consensus state: current_term: 1 leader_uuid: "87285e1acf96486aa31209535a7f6604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } } }
I20260927 00:39:50.972395 10692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:50.972699 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87285e1acf96486aa31209535a7f6604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87285e1acf96486aa31209535a7f6604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } } }
I20260927 00:39:50.972783 10700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:39:50.972805 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604 [sys.catalog]: This master's current role is: LEADER
I20260927 00:39:50.975692 10700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:39:50.981113 10700 catalog_manager.cc:1347] Generated new cluster ID: e6337fad68184ad881bb127e894c3aa8
I20260927 00:39:50.981217 10700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:39:51.011070 10700 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:39:51.012105 10700 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:39:51.017095 10700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: Generated new TSK 0
I20260927 00:39:51.017886 10700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:39:51.059824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:51.174350 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:51.174630 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:51.174674 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:51.178882 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:51.178954 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:51.179018 10711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:39:51.184517 10711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:51.185592 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:51.186632 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:51.196256 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:51.196417 10711 server_base.cc:1034] running on GCE node
W20260927 00:39:51.196321 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:51.196305 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:51.196959 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:51.197592 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:51.198769 10711 hybrid_clock.cc:648] HybridClock initialized: now 1790469591198740 us; error 41 us; skew 500 ppm
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:51.206416 10711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:39:51.207417 10711 webserver.cc:460] Webserver started at http://127.0.98.1:37259/ using document root <none> and password file <none>
I20260927 00:39:51.207999 10711 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:51.208073 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:51.208336 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:51.210572 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b063e3c44eb4e4e9e354347495d9725"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "fd14f7e0782c006254483002133cc3e3"
server_key_iv: "54f449a0e076029cae14ff1094231e90"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.211189 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b063e3c44eb4e4e9e354347495d9725"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "fd14f7e0782c006254483002133cc3e3"
server_key_iv: "54f449a0e076029cae14ff1094231e90"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.215325 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:39:51.217986 10726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.219292 10711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:39:51.219417 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "5b063e3c44eb4e4e9e354347495d9725"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "fd14f7e0782c006254483002133cc3e3"
server_key_iv: "54f449a0e076029cae14ff1094231e90"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.219524 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:51.231482 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:51.234742 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:51.234925 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:51.235522 10711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:51.236409 10711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:51.236459 10711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.236514 10711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:51.236542 10711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.245527 10711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40849
I20260927 00:39:51.245558 10839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40849 every 8 connection(s)
I20260927 00:39:51.246452 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:39:51.246991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10711
I20260927 00:39:51.247102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:39:51.247399 392 external_mini_cluster.cc:1428] Setting key d73eddca52062a487e621a283916e9c9
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:51.266462 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.248772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44033 (local address 127.0.98.62:46259)
0927 00:39:51.248924 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.248929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.250123 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 00:39:51.251774 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.251779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.251845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.251950 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:51.253375 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.254223 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.255489 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.255868 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.255991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.262133 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:51.262137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:51.262139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:51.262169 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:51.263734 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.264395 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.264399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.264401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.264496 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.265046 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.265050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.265052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.265318 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:51.265423 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.266120 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.266278 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:39:51.267323 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.249334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46259 (local address 127.0.98.1:44033)
0927 00:39:51.249929 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.249995 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.251506 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.252138 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.252146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.252582 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:51.253191 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.253207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.254407 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.254410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.255351 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.255362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.256027 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.256030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.256192 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.256884 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:51.256901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:51.261818 (+ 4917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:51.263907 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.263914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.263929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.264283 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.264617 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.264620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.264622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.264907 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.265445 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:51.265455 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:51.265829 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.266018 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.266316 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":128,"threads_started":1}
I20260927 00:39:51.268787 10840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46259
I20260927 00:39:51.269033 10840 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:51.269629 10840 heartbeater.cc:511] Master 127.0.98.62:46259 requested a full tablet report, sending...
I20260927 00:39:51.271341 10649 ts_manager.cc:194] Registered new tserver with Master: 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849)
I20260927 00:39:51.272850 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:39:51.279533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:51.391142 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:51.391496 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:51.391566 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:51.395808 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:51.395880 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:51.395965 10845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:39:51.401489 10845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:51.402657 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:51.403775 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:39:51.414232 10845 server_base.cc:1034] running on GCE node
W20260927 00:39:51.414139 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:51.414132 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:51.414172 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:51.414925 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:51.415653 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:51.416874 10845 hybrid_clock.cc:648] HybridClock initialized: now 1790469591416844 us; error 45 us; skew 500 ppm
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:51.419931 10845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:39:51.421150 10845 webserver.cc:460] Webserver started at http://127.0.98.2:36397/ using document root <none> and password file <none>
I20260927 00:39:51.421751 10845 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:51.421823 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:51.422077 10845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:51.424554 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "2621b791e78a138dbc7d4f83ce144693"
server_key_iv: "7c2f83fa01ac5eac5736e6cfd0a2d8e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.425182 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "2621b791e78a138dbc7d4f83ce144693"
server_key_iv: "7c2f83fa01ac5eac5736e6cfd0a2d8e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.429355 10845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:39:51.432045 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.433488 10845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:39:51.433614 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "2621b791e78a138dbc7d4f83ce144693"
server_key_iv: "7c2f83fa01ac5eac5736e6cfd0a2d8e7"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.433724 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:51.446244 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:51.450096 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:51.450305 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:51.450986 10845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:51.451908 10845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:51.451958 10845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.452013 10845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:51.452040 10845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.460963 10845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33827
I20260927 00:39:51.460985 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33827 every 8 connection(s)
I20260927 00:39:51.461947 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:39:51.466389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10845
I20260927 00:39:51.466477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:39:51.466722 392 external_mini_cluster.cc:1428] Setting key 0c0b9dbbcda039a7965765a9e43e6cb9
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:39:51.477183 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.463785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41929 (local address 127.0.98.62:46259)
0927 00:39:51.463909 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.463914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.464834 (+ 920us) server_negotiation.cc:407] Connection header received
0927 00:39:51.466187 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.466191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.466250 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.466323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:51.467625 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.468290 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.469936 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.470160 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.470251 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.473455 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:51.473460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:51.473463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:51.473496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:51.474977 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.475543 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.475547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.475549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.475647 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.475968 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.475972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.475974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.476258 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:51.476355 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.476909 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.477024 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:39:51.477973 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.464179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46259 (local address 127.0.98.2:41929)
0927 00:39:51.464668 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.464721 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.465974 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.466540 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.466547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.466932 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:51.467464 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.467484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.468416 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.468419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.469813 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.469825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.470307 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.470312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.470490 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.471098 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:51.471119 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:51.473236 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:51.475117 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.475122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.475137 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.475431 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.475738 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.475741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.475742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.475885 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.476349 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:51.476355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:51.476655 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.476823 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.477097 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:39:51.479494 10974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46259
I20260927 00:39:51.479763 10974 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:51.480338 10974 heartbeater.cc:511] Master 127.0.98.62:46259 requested a full tablet report, sending...
I20260927 00:39:51.481508 10649 ts_manager.cc:194] Registered new tserver with Master: 0ce95cb2f7ea48ecbdef29e7ab40f3b9 (127.0.98.2:33827)
I20260927 00:39:51.482074 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:39:51.495332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:39:51.610419 10979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:39:51.610685 10979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:39:51.610760 10979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:39:51.615267 10979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:39:51.615365 10979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:39:51.615455 10979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:39:51.621093 10979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:39:51.622221 10979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:39:51.623266 10979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:39:51.633307 10984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:51.633360 10987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:39:51.633317 10985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:39:51.633541 10979 server_base.cc:1034] running on GCE node
I20260927 00:39:51.633970 10979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:39:51.634609 10979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:39:51.635782 10979 hybrid_clock.cc:648] HybridClock initialized: now 1790469591635762 us; error 46 us; skew 500 ppm
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:39:51.638696 10979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:39:51.639706 10979 webserver.cc:460] Webserver started at http://127.0.98.3:39049/ using document root <none> and password file <none>
I20260927 00:39:51.640305 10979 fs_manager.cc:359] Metadata directory not provided
I20260927 00:39:51.640373 10979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:39:51.640568 10979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:39:51.642688 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "78c3cd38f1994e438e81bbf4ba03c896"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "1187b71a27eb0c8a740ac4fb79b79426"
server_key_iv: "23dbec0e3f1f3495f42e6786499ced7a"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.643231 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78c3cd38f1994e438e81bbf4ba03c896"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "1187b71a27eb0c8a740ac4fb79b79426"
server_key_iv: "23dbec0e3f1f3495f42e6786499ced7a"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.647022 10979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:39:51.649490 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.650733 10979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:39:51.650928 10979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "78c3cd38f1994e438e81bbf4ba03c896"
format_stamp: "Formatted at 2026-09-27 00:39:51 on dist-test-slave-xltq"
server_key: "1187b71a27eb0c8a740ac4fb79b79426"
server_key_iv: "23dbec0e3f1f3495f42e6786499ced7a"
server_key_version: "encryptionkey@0"
I20260927 00:39:51.651038 10979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:39:51.666195 10979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:39:51.669322 10979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:39:51.669499 10979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:39:51.670074 10979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:39:51.670933 10979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:39:51.670979 10979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.671020 10979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:39:51.671039 10979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:39:51.679761 10979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42137
I20260927 00:39:51.679807 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42137 every 8 connection(s)
I20260927 00:39:51.680794 10979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:39:51.681864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 10979
I20260927 00:39:51.681960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:39:51.682264 392 external_mini_cluster.cc:1428] Setting key 3bad9d300dc126a05e20eed1539dbe0c
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:39:51.695048 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.682901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36221 (local address 127.0.98.62:46259)
0927 00:39:51.683043 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.683047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.683876 (+ 829us) server_negotiation.cc:407] Connection header received
0927 00:39:51.685231 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.685233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.685277 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.685343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:51.686453 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.687039 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.688162 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.688431 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.688507 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.690971 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:51.690974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:51.690976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:51.691004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:51.692683 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.693335 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.693339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.693340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.693425 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.693863 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.693866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.693867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.694079 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:51.694184 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.694740 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.694880 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:39:51.696000 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.683198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46259 (local address 127.0.98.3:36221)
0927 00:39:51.683712 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.683773 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.685049 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.685481 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.685488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.685836 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:39:51.686327 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.686341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.687154 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.687157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.688071 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.688080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.688556 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.688559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.688701 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.689267 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:39:51.689283 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:39:51.690747 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:39:51.692841 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.692847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.692864 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.693204 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.693515 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:39:51.693517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:39:51.693519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:39:51.693671 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:39:51.694194 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:39:51.694200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:39:51.694484 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.694668 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.695052 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260927 00:39:51.697053 11108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46259
I20260927 00:39:51.697274 11108 heartbeater.cc:464] Registering TS with master...
I20260927 00:39:51.697757 11108 heartbeater.cc:511] Master 127.0.98.62:46259 requested a full tablet report, sending...
I20260927 00:39:51.698850 10649 ts_manager.cc:194] Registered new tserver with Master: 78c3cd38f1994e438e81bbf4ba03c896 (127.0.98.3:42137)
I20260927 00:39:51.699453 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36221
I20260927 00:39:51.708060 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:39:51.717428 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.709987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.62:46259)
0927 00:39:51.710148 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.710154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.710367 (+ 213us) server_negotiation.cc:407] Connection header received
0927 00:39:51.710485 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.710488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.710533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.710600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:39:51.711390 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.711898 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.712824 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.713041 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.713131 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.714445 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:39:51.714448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:39:51.714451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:39:51.714512 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:39:51.715804 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.716196 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.716217 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.716220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.716314 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:39:51.716560 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:39:51.716563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:39:51.716565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:39:51.716796 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:39:51.716914 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.717146 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.717275 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:39:51.720593 10649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:39:51.723637 10649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:39:51.739017 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.733610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42137 (local address 127.0.0.1:35816)
0927 00:39:51.734660 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.734671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.734741 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.735354 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.735356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.735370 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.735439 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.735443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.736469 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.736472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.737515 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.737525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.738379 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.738382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.738509 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.738616 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.738725 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.738861 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260927 00:39:51.739025 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.733787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.0.98.3:42137)
0927 00:39:51.734230 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.734240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.734829 (+ 589us) server_negotiation.cc:407] Connection header received
0927 00:39:51.734909 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.734914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.735085 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.735227 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.735546 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.736349 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.737645 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.738257 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.738360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.738525 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.738768 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.738873 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":150,"threads_started":1}
I20260927 00:39:51.739475 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.733860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.0.98.1:40849)
0927 00:39:51.734251 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.734264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.734527 (+ 263us) server_negotiation.cc:407] Connection header received
0927 00:39:51.734616 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.734621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.734823 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.734977 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.735204 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.735977 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.737733 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.738668 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.738771 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.738923 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.739224 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.739342 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260927 00:39:51.739897 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.732989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40849 (local address 127.0.0.1:51704)
0927 00:39:51.734401 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.734414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.734507 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.734959 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.734963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.734981 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.735092 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.735098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.736106 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.736109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.737593 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.737602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.738977 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.738981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.739067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.739089 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.739671 (+ 582us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.739750 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":92,"threads_started":1}
I20260927 00:39:51.742095 11042 tablet_service.cc:1467] Processing CreateTablet for tablet ff248487bcff40b1b38c0f595632eaae (DEFAULT_TABLE table=test-table [id=5603a468a1ff4e4dbb4ec77710121833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:51.742473 10774 tablet_service.cc:1467] Processing CreateTablet for tablet ff248487bcff40b1b38c0f595632eaae (DEFAULT_TABLE table=test-table [id=5603a468a1ff4e4dbb4ec77710121833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:51.743237 11042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff248487bcff40b1b38c0f595632eaae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:51.743455 10774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff248487bcff40b1b38c0f595632eaae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:51.743746 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.732958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33827 (local address 127.0.0.1:49694)
0927 00:39:51.733874 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.733935 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.734079 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.734909 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.734915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.734937 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.735086 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.735093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.737537 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.737540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.738751 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.738771 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.743311 (+ 4540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.743315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.743425 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.743451 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.743522 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.743603 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260927 00:39:51.744333 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.733137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.98.2:33827)
0927 00:39:51.733464 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.733477 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.734224 (+ 747us) server_negotiation.cc:407] Connection header received
0927 00:39:51.734314 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.734320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.734549 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.734696 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.736134 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.737344 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.738893 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.742956 (+ 4063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.743074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.743224 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.744041 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.744150 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260927 00:39:51.747335 10908 tablet_service.cc:1467] Processing CreateTablet for tablet ff248487bcff40b1b38c0f595632eaae (DEFAULT_TABLE table=test-table [id=5603a468a1ff4e4dbb4ec77710121833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:39:51.748534 10908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff248487bcff40b1b38c0f595632eaae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:39:51.750356 11125 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Bootstrap starting.
I20260927 00:39:51.751958 11126 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: Bootstrap starting.
I20260927 00:39:51.753190 11125 tablet_bootstrap.cc:654] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:51.754300 11125 log.cc:824] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:51.754398 11126 tablet_bootstrap.cc:654] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:51.755613 11126 log.cc:824] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:51.756458 11125 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: No bootstrap required, opened a new log
I20260927 00:39:51.756664 11125 ts_tablet_manager.cc:1397] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:39:51.757712 11126 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: No bootstrap required, opened a new log
I20260927 00:39:51.757905 11126 ts_tablet_manager.cc:1397] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:39:51.759948 11125 raft_consensus.cc:348] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.760247 11125 raft_consensus.cc:374] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:51.760329 11125 raft_consensus.cc:729] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c3cd38f1994e438e81bbf4ba03c896, State: Initialized, Role: FOLLOWER
I20260927 00:39:51.760855 11125 consensus_queue.cc:260] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.760932 11126 raft_consensus.cc:348] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.761183 11126 raft_consensus.cc:374] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:51.761245 11126 raft_consensus.cc:729] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b063e3c44eb4e4e9e354347495d9725, State: Initialized, Role: FOLLOWER
I20260927 00:39:51.761602 11108 heartbeater.cc:503] Master 127.0.98.62:46259 was elected leader, sending a full tablet report...
I20260927 00:39:51.761777 11126 consensus_queue.cc:260] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.761940 11125 ts_tablet_manager.cc:1428] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:39:51.762444 10840 heartbeater.cc:503] Master 127.0.98.62:46259 was elected leader, sending a full tablet report...
I20260927 00:39:51.762786 11126 ts_tablet_manager.cc:1428] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 00:39:51.763191 11129 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Bootstrap starting.
I20260927 00:39:51.766279 11129 tablet_bootstrap.cc:654] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Neither blocks nor log segments found. Creating new log.
I20260927 00:39:51.767342 11129 log.cc:824] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:39:51.769513 11129 tablet_bootstrap.cc:492] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: No bootstrap required, opened a new log
I20260927 00:39:51.769708 11129 ts_tablet_manager.cc:1397] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 00:39:51.773166 11129 raft_consensus.cc:348] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.773574 11129 raft_consensus.cc:374] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:39:51.773689 11129 raft_consensus.cc:729] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce95cb2f7ea48ecbdef29e7ab40f3b9, State: Initialized, Role: FOLLOWER
I20260927 00:39:51.774231 11129 consensus_queue.cc:260] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.775094 10974 heartbeater.cc:503] Master 127.0.98.62:46259 was elected leader, sending a full tablet report...
I20260927 00:39:51.775471 11129 ts_tablet_manager.cc:1428] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 00:39:51.833808 11133 raft_consensus.cc:482] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:51.834061 11130 raft_consensus.cc:482] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:39:51.834057 11133 raft_consensus.cc:504] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.834257 11130 raft_consensus.cc:504] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.835283 11130 leader_election.cc:288] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce95cb2f7ea48ecbdef29e7ab40f3b9 (127.0.98.2:33827), 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849)
I20260927 00:39:51.836428 11133 leader_election.cc:288] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849), 78c3cd38f1994e438e81bbf4ba03c896 (127.0.98.3:42137)
I20260927 00:39:51.840021 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42137 (local address 127.0.98.2:58203)
0927 00:39:51.835701 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.835717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.835804 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.836166 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.836169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.836188 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.836337 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.836344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837347 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.837351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.838632 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.838643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.839565 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.839676 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.839700 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.839770 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.839853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 00:39:51.840241 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40849 (local address 127.0.98.3:44949)
0927 00:39:51.835987 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.835998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.836056 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.836551 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.836554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.836569 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.836659 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.836667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837733 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.837737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.839006 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.839016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.839844 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.839955 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.839977 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.840026 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.840096 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":7,"spinlock_wait_cycles":896}
I20260927 00:39:51.840553 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33827 (local address 127.0.98.3:57921)
0927 00:39:51.836152 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.836165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.836330 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.836654 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.836656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.836674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.836792 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.836797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837736 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.837739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.839102 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.839113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.840119 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.840122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.840243 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.840266 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.840340 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.840404 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":152,"thread_start_us":83,"threads_started":1}
I20260927 00:39:51.840762 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44949 (local address 127.0.98.1:40849)
0927 00:39:51.836331 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.836336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.836349 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:51.836397 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.836400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.836446 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.836772 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.836886 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837611 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839130 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.839726 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.840466 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.840525 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.840560 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.840631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":75,"threads_started":1}
I20260927 00:39:51.840850 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58203 (local address 127.0.98.3:42137)
0927 00:39:51.835758 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.835762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.835775 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:51.835922 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.835925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.835972 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.836037 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.836475 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837224 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.838803 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.839340 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839424 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.839508 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.840688 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.840751 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37,"spinlock_wait_cycles":6400}
I20260927 00:39:51.840969 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57921 (local address 127.0.98.2:33827)
0927 00:39:51.836081 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.836085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.836262 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:39:51.836440 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.836443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.836485 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.836532 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.836899 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.837623 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839228 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.839897 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.839992 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.840050 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.840774 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.840844 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":41}
I20260927 00:39:51.841427 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c3cd38f1994e438e81bbf4ba03c896" is_pre_election: true
I20260927 00:39:51.841692 11062 raft_consensus.cc:2454] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce95cb2f7ea48ecbdef29e7ab40f3b9 in term 0.
I20260927 00:39:51.841720 10794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "78c3cd38f1994e438e81bbf4ba03c896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b063e3c44eb4e4e9e354347495d9725" is_pre_election: true
I20260927 00:39:51.841869 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "78c3cd38f1994e438e81bbf4ba03c896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" is_pre_election: true
I20260927 00:39:51.842006 10794 raft_consensus.cc:2454] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c3cd38f1994e438e81bbf4ba03c896 in term 0.
I20260927 00:39:51.842069 10928 raft_consensus.cc:2454] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c3cd38f1994e438e81bbf4ba03c896 in term 0.
I20260927 00:39:51.842195 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56641 (local address 127.0.98.1:40849)
0927 00:39:51.835902 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.835906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.837086 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 00:39:51.837246 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.837250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.837292 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.837334 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:39:51.837735 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.838492 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.840024 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.840611 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.841240 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.841358 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.842019 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.842098 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260927 00:39:51.842200 10862 leader_election.cc:302] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ce95cb2f7ea48ecbdef29e7ab40f3b9, 78c3cd38f1994e438e81bbf4ba03c896; no voters:
I20260927 00:39:51.842507 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.835510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40849 (local address 127.0.98.2:56641)
0927 00:39:51.837010 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.837022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:39:51.837146 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:39:51.837441 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:39:51.837444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:39:51.837458 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:39:51.837622 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.837627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.838689 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.838692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.839911 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:39:51.839918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.841757 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.841760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:39:51.841868 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.841890 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:39:51.842306 (+ 416us) client_negotiation.cc:240] Negotiation successful
0927 00:39:51.842382 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"mutex_wait_us":700,"thread_start_us":158,"threads_started":1}
I20260927 00:39:51.842489 10995 leader_election.cc:302] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ce95cb2f7ea48ecbdef29e7ab40f3b9, 5b063e3c44eb4e4e9e354347495d9725, 78c3cd38f1994e438e81bbf4ba03c896; no voters:
I20260927 00:39:51.842576 11133 raft_consensus.cc:2781] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:51.842729 11130 raft_consensus.cc:2781] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:39:51.842809 11130 raft_consensus.cc:482] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:51.842883 11130 raft_consensus.cc:3037] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:51.842904 10794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b063e3c44eb4e4e9e354347495d9725" is_pre_election: true
I20260927 00:39:51.843029 10794 raft_consensus.cc:2454] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ce95cb2f7ea48ecbdef29e7ab40f3b9 in term 0.
I20260927 00:39:51.842657 11133 raft_consensus.cc:482] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:39:51.843314 11133 raft_consensus.cc:3037] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:51.844291 11130 raft_consensus.cc:504] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.844472 11133 raft_consensus.cc:504] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.844688 11130 leader_election.cc:288] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [CANDIDATE]: Term 1 election: Requested vote from peers 0ce95cb2f7ea48ecbdef29e7ab40f3b9 (127.0.98.2:33827), 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849)
I20260927 00:39:51.844797 11133 leader_election.cc:288] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [CANDIDATE]: Term 1 election: Requested vote from peers 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849), 78c3cd38f1994e438e81bbf4ba03c896 (127.0.98.3:42137)
I20260927 00:39:51.845022 10928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "78c3cd38f1994e438e81bbf4ba03c896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9"
I20260927 00:39:51.845022 10794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "78c3cd38f1994e438e81bbf4ba03c896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b063e3c44eb4e4e9e354347495d9725"
I20260927 00:39:51.845098 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b063e3c44eb4e4e9e354347495d9725"
I20260927 00:39:51.845127 10794 raft_consensus.cc:3037] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:39:51.845183 10928 raft_consensus.cc:2379] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78c3cd38f1994e438e81bbf4ba03c896 in current term 1: Already voted for candidate 0ce95cb2f7ea48ecbdef29e7ab40f3b9 in this term.
I20260927 00:39:51.845402 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff248487bcff40b1b38c0f595632eaae" candidate_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c3cd38f1994e438e81bbf4ba03c896"
I20260927 00:39:51.845568 11062 raft_consensus.cc:2379] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ce95cb2f7ea48ecbdef29e7ab40f3b9 in current term 1: Already voted for candidate 78c3cd38f1994e438e81bbf4ba03c896 in this term.
I20260927 00:39:51.845888 10862 leader_election.cc:302] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ce95cb2f7ea48ecbdef29e7ab40f3b9; no voters: 5b063e3c44eb4e4e9e354347495d9725, 78c3cd38f1994e438e81bbf4ba03c896
I20260927 00:39:51.846073 11133 raft_consensus.cc:2726] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 00:39:51.846369 10794 raft_consensus.cc:2454] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c3cd38f1994e438e81bbf4ba03c896 in term 1.
I20260927 00:39:51.846606 10995 leader_election.cc:302] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b063e3c44eb4e4e9e354347495d9725, 78c3cd38f1994e438e81bbf4ba03c896; no voters: 0ce95cb2f7ea48ecbdef29e7ab40f3b9
I20260927 00:39:51.846886 11130 raft_consensus.cc:2781] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:39:51.847080 11130 raft_consensus.cc:686] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 LEADER]: Becoming Leader. State: Replica: 78c3cd38f1994e438e81bbf4ba03c896, State: Running, Role: LEADER
I20260927 00:39:51.847397 11130 consensus_queue.cc:237] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } }
I20260927 00:39:51.850838 10648 catalog_manager.cc:5495] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c3cd38f1994e438e81bbf4ba03c896 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "78c3cd38f1994e438e81bbf4ba03c896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c3cd38f1994e438e81bbf4ba03c896" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42137 } health_report { overall_health: HEALTHY } } }
I20260927 00:39:51.867202 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.862895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.98.3:42137)
0927 00:39:51.863017 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:39:51.863020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:39:51.863033 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:39:51.863157 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:39:51.863159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:39:51.863213 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:39:51.863303 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:39:51.863606 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.864154 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.865412 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:39:51.865790 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:39:51.865901 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:39:51.866173 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:39:51.866737 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:39:51.866908 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 00:39:51.867025 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 00:39:51.867074 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:39:51.873468 10794 raft_consensus.cc:1261] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 1 FOLLOWER]: Refusing update from remote peer 78c3cd38f1994e438e81bbf4ba03c896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:51.873469 10928 raft_consensus.cc:1261] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Refusing update from remote peer 78c3cd38f1994e438e81bbf4ba03c896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:39:51.874092 11130 consensus_queue.cc:1035] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce95cb2f7ea48ecbdef29e7ab40f3b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:51.874539 11137 consensus_queue.cc:1035] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b063e3c44eb4e4e9e354347495d9725" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:39:51.883801 11140 mvcc.cc:205] Tried to move back new op lower bound from 7333763448305840128 to 7333763448209457152. Current Snapshot: MvccSnapshot[applied={T|T < 7333763448305840128 or (T in {7333763448305840128})}]
I20260927 00:39:51.884121 11141 mvcc.cc:205] Tried to move back new op lower bound from 7333763448305840128 to 7333763448209457152. Current Snapshot: MvccSnapshot[applied={T|T < 7333763448305840128}]
I20260927 00:39:51.890055 10648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:51.890267 10648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:39:51.893018 10648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87285e1acf96486aa31209535a7f6604: Sending DeleteTablet for 3 replicas of tablet ff248487bcff40b1b38c0f595632eaae
I20260927 00:39:51.893929 10908 tablet_service.cc:1514] Processing DeleteTablet for tablet ff248487bcff40b1b38c0f595632eaae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49694
I20260927 00:39:51.894053 10774 tablet_service.cc:1514] Processing DeleteTablet for tablet ff248487bcff40b1b38c0f595632eaae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51704
I20260927 00:39:51.894582 11150 tablet_replica.cc:330] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725: stopping tablet replica
I20260927 00:39:51.894631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10711
I20260927 00:39:51.894729 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet ff248487bcff40b1b38c0f595632eaae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:39:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 00:39:51.895002 11150 raft_consensus.cc:2229] T ff248487bcff40b1b38c0f595632eaae P 5b063e3c44eb4e4e9e354347495d9725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:51.895380 11151 tablet_replica.cc:330] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: stopping tablet replica
I20260927 00:39:51.895678 11151 raft_consensus.cc:2229] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:39:51.895978 11151 raft_consensus.cc:2258] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:51.896692 11149 tablet_replica.cc:330] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: stopping tablet replica
I20260927 00:39:51.896994 11149 raft_consensus.cc:2229] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:39:51.897306 11149 raft_consensus.cc:2258] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:39:51.897485 11151 ts_tablet_manager.cc:1905] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:51.898873 11149 ts_tablet_manager.cc:1905] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:39:51.900345 11151 ts_tablet_manager.cc:1918] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:51.900436 11151 log.cc:1196] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/ff248487bcff40b1b38c0f595632eaae
I20260927 00:39:51.900789 11151 ts_tablet_manager.cc:1939] T ff248487bcff40b1b38c0f595632eaae P 78c3cd38f1994e438e81bbf4ba03c896: Deleting consensus metadata
I20260927 00:39:51.901886 11149 ts_tablet_manager.cc:1918] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:39:51.902030 11149 log.cc:1196] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/ff248487bcff40b1b38c0f595632eaae
I20260927 00:39:51.902308 11149 ts_tablet_manager.cc:1939] T ff248487bcff40b1b38c0f595632eaae P 0ce95cb2f7ea48ecbdef29e7ab40f3b9: Deleting consensus metadata
I20260927 00:39:51.903445 10636 catalog_manager.cc:4841] TS 0ce95cb2f7ea48ecbdef29e7ab40f3b9 (127.0.98.2:33827): tablet ff248487bcff40b1b38c0f595632eaae (table test-table [id=5603a468a1ff4e4dbb4ec77710121833]) successfully deleted
I20260927 00:39:51.903451 10634 catalog_manager.cc:4841] TS 78c3cd38f1994e438e81bbf4ba03c896 (127.0.98.3:42137): tablet ff248487bcff40b1b38c0f595632eaae (table test-table [id=5603a468a1ff4e4dbb4ec77710121833]) successfully deleted
W20260927 00:39:51.904532 10633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40849 (error 108)
I20260927 00:39:51.905345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10845
I20260927 00:39:51.905565 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:39:51.905239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40849 (local address 127.0.0.1:51716)
0927 00:39:51.905407 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:39:51.905477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 00:39:51.906248 10633 catalog_manager.cc:4584] TS 5b063e3c44eb4e4e9e354347495d9725 (127.0.98.1:40849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff248487bcff40b1b38c0f595632eaae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40849: connect: Connection refused (error 111)
I20260927 00:39:51.911262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10979
I20260927 00:39:51.916338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 10617
2026-09-27T00:39:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2238 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[11155](info): set up 2 sockets
Sep 27 00:39:51 dist-test-slave-xltq krb5kdc[11155](info): commencing operation
krb5kdc: starting...
W20260927 00:39:52.881266 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:39:52 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:39:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:39:53.056444 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSbnIrP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:39:55.886 UTC [11185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:39:55.886 UTC [11185] LOG: listening on IPv4 address "127.0.98.20", port 44969
2026-09-27 00:39:55.891 UTC [11185] LOG: listening on Unix socket "/tmp/.s.PGSQL.44969"
2026-09-27 00:39:55.915 UTC [11188] LOG: database system was shut down at 2026-09-27 00:39:54 UTC
2026-09-27 00:39:55.919 UTC [11185] LOG: database system is ready to accept connections
I20260927 00:39:56.797060 392 mini_postgres.cc:96] Postgres bound to 44969
2026-09-27 00:39:56.803 UTC [11196] FATAL: database "slave" does not exist
127.0.98.20:44969 - accepting connections
I20260927 00:39:56.804225 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:39:56.820137 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:39:57.370175 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:39:57.370256 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin
2026-09-27 00:39:57,667 [I] DB FLAVOR :POSTGRES
2026-09-27 00:39:57,667 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:39:57,667 [I] Checking connection..
2026-09-27 00:39:57,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:39:57,942 [I] Checking connection passed.
2026-09-27 00:39:57,943 [I] --------- Verifying version history table ---------
2026-09-27 00:39:57,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:39:58,217 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:39:58,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:39:58,474 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:39:58,475 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:39:58,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:39:58.743 UTC [11260] WARNING: there is no transaction in progress
2026-09-27 00:39:58,748 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:39:58,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:39:58,988 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:39:58,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:39:58,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:39:59,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:39:59,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xltq.c.gcp-upstream.internal','N') ;"
2026-09-27 00:39:59,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:39:59,973 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:39:59,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:40:00,227 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:40:00,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:40:00,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:40:00,740 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:40:00,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:40:01.846 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:01.865 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:01.887 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:01.904 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:01.924 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:01.948 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.015 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.023 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.040 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.053 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.065 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.082 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.094 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.102 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.109 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02.117 UTC [11368] WARNING: there is no transaction in progress
2026-09-27 00:40:02,916 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:40:02,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:40:03,153 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:40:03,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:40:03,402 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:40:03,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:40:03,653 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:40:03,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:40:03,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:40:04,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xltq.c.gcp-upstream.internal';"
2026-09-27 00:40:04,413 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:40:04,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:40:04,653 [I] DB_PATCHES have already been applied
I20260927 00:40:04.659664 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSbnIrP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:40:04.659736 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:40:04.662999 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36051
Sep 27, 2026 12:40:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:40:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:40:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:40:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:40:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:40:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:40:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:40:05.671629 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.867s user 0.004s sys 0.006s
Sep 27, 2026 12:40:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:40:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:40:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:40:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:40:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:40:28.936465 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:40:29 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469629, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:40:29.034511 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:40:29.044440 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:40:30.107981 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:40:30.108078 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms
I20260927 00:40:30.108152 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44969
2026-09-27 00:40:30,303 [I] DB FLAVOR :POSTGRES
2026-09-27 00:40:30,303 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:40:30,303 [I] Checking connection
2026-09-27 00:40:30,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:40:30,561 [I] connection success
2026-09-27 00:40:30,561 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:40:30,561 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:40:30,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:40:30,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:40:30,822 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:40:30,822 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:40:30,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:40:31,099 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:40:31.645131 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:40:31.860944 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:40:31.931355 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:40:31 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469629, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:32.400718 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:40:32.400876 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:40:32.400897 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:40:32.405081 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11549
I20260927 00:40:32.405206 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39633
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:40:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:40:34.503607 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.469s user 0.010s sys 0.167s
I20260927 00:40:34.503736 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:40:34.503852 392 mini_ranger_kms.cc:401] 127.0.98.20:39633/kms/v1/keys
Sep 27 00:40:34 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:40:35 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:40:35 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469634, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:40:38 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:40:38.120628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40239
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39633/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:40:38.240433 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:40:38.240744 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:40:38.240823 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:40:38.245231 11599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:40:38.245311 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:40:38.245329 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:40:38.245345 11599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:40:38.245359 11599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:40:38.251307 11599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39633/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40239
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:40:38.252571 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:40:38.253666 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:40:38.263499 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:38.263510 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:38.263510 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:40:38.263661 11599 server_base.cc:1034] running on GCE node
I20260927 00:40:38.264539 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:40:38.265838 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:40:38.267053 11599 hybrid_clock.cc:648] HybridClock initialized: now 1790469638267033 us; error 36 us; skew 500 ppm
Sep 27 00:40:38 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:40:38.270701 11599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:40:38.271780 11599 webserver.cc:460] Webserver started at http://127.0.98.62:45957/ using document root <none> and password file <none>
I20260927 00:40:38.272392 11599 fs_manager.cc:359] Metadata directory not provided
I20260927 00:40:38.272456 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:40:38.272687 11599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:40:38 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:38.332903 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "d3a02b8cecb54856964b33093902eb43"
format_stamp: "Formatted at 2026-09-27 00:40:38 on dist-test-slave-xltq"
server_key: "06982fe5e91c5ccae25151d748b4a4b0"
server_key_iv: "26f48aad12e072aee0e52831eade6f43"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:38.333657 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3a02b8cecb54856964b33093902eb43"
format_stamp: "Formatted at 2026-09-27 00:40:38 on dist-test-slave-xltq"
server_key: "06982fe5e91c5ccae25151d748b4a4b0"
server_key_iv: "26f48aad12e072aee0e52831eade6f43"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:38.337893 11599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:40:38.357247 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:38.358716 11599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:40:38.358850 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "d3a02b8cecb54856964b33093902eb43"
format_stamp: "Formatted at 2026-09-27 00:40:38 on dist-test-slave-xltq"
server_key: "06982fe5e91c5ccae25151d748b4a4b0"
server_key_iv: "26f48aad12e072aee0e52831eade6f43"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:38.358987 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:40:38.373724 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:40:38.376899 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:40:38.377094 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:40:38.385877 11599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40239
I20260927 00:40:38.385905 11666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40239 every 8 connection(s)
I20260927 00:40:38.386974 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:40:38.389240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 11599
I20260927 00:40:38.389369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:40:38.389647 392 external_mini_cluster.cc:1428] Setting key 2cb205cfc33676e0c87b7bfd629e8e9a
I20260927 00:40:38.390141 11667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:40:38.397454 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: Bootstrap starting.
Sep 27 00:40:38 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:40:38.399892 11667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: Neither blocks nor log segments found. Creating new log.
I20260927 00:40:38.400815 11667 log.cc:824] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: Log is configured to *not* fsync() on all Append() calls
I20260927 00:40:38.402848 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: No bootstrap required, opened a new log
I20260927 00:40:38.405853 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:38.391220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.0.98.62:40239)
0927 00:40:38.391688 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:38.391708 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:38.391750 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:40:38.392585 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:38.392611 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:38.393070 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:38.393469 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:40:38.394314 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:38.395755 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:38.396952 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:38.397672 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:38.397829 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:38.400366 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:40:38.400372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:40:38.400388 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:40:38.400420 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:40:38.402635 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:38.403188 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:38.403193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:38.403198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:38.403322 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:38.403721 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:38.403724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:38.403726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:38.404170 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:40:38.404394 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:38.404704 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 00:40:38.404999 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260927 00:40:38.406705 11667 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260927 00:40:38.406975 11667 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:40:38.407024 11667 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a02b8cecb54856964b33093902eb43, State: Initialized, Role: FOLLOWER
I20260927 00:40:38.407506 11667 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260927 00:40:38.407682 11667 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:40:38.407744 11667 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:40:38.407838 11667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:40:38.408942 11667 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260927 00:40:38.409253 11667 leader_election.cc:302] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a02b8cecb54856964b33093902eb43; no voters:
I20260927 00:40:38.409551 11667 leader_election.cc:288] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:40:38.409643 11672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:40:38.409880 11672 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [term 1 LEADER]: Becoming Leader. State: Replica: d3a02b8cecb54856964b33093902eb43, State: Running, Role: LEADER
I20260927 00:40:38.410224 11672 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } }
I20260927 00:40:38.410856 11667 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:40:38.411892 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a02b8cecb54856964b33093902eb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260927 00:40:38.412034 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [sys.catalog]: This master's current role is: LEADER
I20260927 00:40:38.412336 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a02b8cecb54856964b33093902eb43. Latest consensus state: current_term: 1 leader_uuid: "d3a02b8cecb54856964b33093902eb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a02b8cecb54856964b33093902eb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40239 } } }
I20260927 00:40:38.412429 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43 [sys.catalog]: This master's current role is: LEADER
I20260927 00:40:38.412945 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:40:38.416389 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:40:38.416898 11667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:40:38.424875 11681 catalog_manager.cc:1347] Generated new cluster ID: 65df04dda2b64326ac1f44b131f52e90
I20260927 00:40:38.424964 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:40:38.431730 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:40:38.432796 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:40:38.439105 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: Generated new TSK 0
I20260927 00:40:38.439810 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:40:39.177639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:40:39.301620 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:40:39.301911 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:40:39.301973 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:40:39.306425 11712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:40:39.306492 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:40:39.306556 11712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:40:39.312741 11712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:40:39.313925 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:40:39.314961 11712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:40:39.324659 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:39.324627 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:39.324627 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:40:39.324856 11712 server_base.cc:1034] running on GCE node
I20260927 00:40:39.325464 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:40:39.326339 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:40:39.327553 11712 hybrid_clock.cc:648] HybridClock initialized: now 1790469639327537 us; error 32 us; skew 500 ppm
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:40:39.332288 11712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:40:39.333679 11712 webserver.cc:460] Webserver started at http://127.0.98.1:40307/ using document root <none> and password file <none>
I20260927 00:40:39.334547 11712 fs_manager.cc:359] Metadata directory not provided
I20260927 00:40:39.334718 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:40:39.335043 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:39.354820 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "0316eb8a129f466fbf40b0f62ab06f05"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "8425a9bc54313515961c4c09b2d5b7ad"
server_key_iv: "86a218ed49073438565f8064b091b7ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.355618 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0316eb8a129f466fbf40b0f62ab06f05"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "8425a9bc54313515961c4c09b2d5b7ad"
server_key_iv: "86a218ed49073438565f8064b091b7ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.370692 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.007s sys 0.001s
I20260927 00:40:39.392902 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:39.394495 11712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:40:39.394632 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "0316eb8a129f466fbf40b0f62ab06f05"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "8425a9bc54313515961c4c09b2d5b7ad"
server_key_iv: "86a218ed49073438565f8064b091b7ac"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.394776 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:40:39.417279 11712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:40:39.421084 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:40:39.421340 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:40:39.422107 11712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:40:39.423216 11712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:40:39.423276 11712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:39.423332 11712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:40:39.423388 11712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 00:40:39.438846 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35247
I20260927 00:40:39.440094 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:40:39.447300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 11712
I20260927 00:40:39.447453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:40:39.447788 392 external_mini_cluster.cc:1428] Setting key ae0f83967e1b1f3fbc36662398ff9d87
I20260927 00:40:39.451589 11842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35247 every 8 connection(s)
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:40:39.472298 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:39.443669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33607 (local address 127.0.98.62:40239)
0927 00:40:39.452371 (+ 8702us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:39.452376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:39.452393 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:40:39.453846 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:39.453850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:39.453920 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:39.454013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:40:39.455898 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.456762 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.458490 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.458876 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.458999 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:39.464676 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:40:39.464680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:40:39.464683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:40:39.464716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:40:39.466772 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:39.467520 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:39.467524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:39.467526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:39.467645 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:39.470723 (+ 3078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:39.470728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:39.470730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:39.471015 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:40:39.471098 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:39.471928 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 00:40:39.472074 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8617,"thread_start_us":78,"threads_started":1}
I20260927 00:40:39.473600 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:39.444245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.1:33607)
0927 00:40:39.452172 (+ 7927us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:39.452288 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:39.453533 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:39.454489 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:39.454501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:39.454900 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:40:39.455705 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:39.455728 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.456929 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.456932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:39.458322 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:39.458335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.459212 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.459216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:39.459432 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:39.462545 (+ 3113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:40:39.462571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:40:39.464418 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:40:39.466974 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:39.466982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:39.467003 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:39.467390 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:39.467764 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:39.467767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:39.467769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:39.468073 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:39.471197 (+ 3124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:40:39.471205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:40:39.471611 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 00:40:39.471833 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 00:40:39.472224 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7617,"thread_start_us":126,"threads_started":1}
I20260927 00:40:39.475826 11843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260927 00:40:39.476095 11843 heartbeater.cc:464] Registering TS with master...
I20260927 00:40:39.476818 11843 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260927 00:40:39.478657 11631 ts_manager.cc:194] Registered new tserver with Master: 0316eb8a129f466fbf40b0f62ab06f05 (127.0.98.1:35247)
I20260927 00:40:39.480408 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:40:39.505798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:40:39.678337 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:40:39.678764 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:40:39.678848 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:40:39.685418 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:40:39.685519 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:40:39.685611 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:40:39.692503 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:40:39.693728 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:40:39.694834 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:40:39.706459 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:39.706543 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:40:39.706826 11851 server_base.cc:1034] running on GCE node
W20260927 00:40:39.706461 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:40:39.707409 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:40:39.708186 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:40:39.709429 11851 hybrid_clock.cc:648] HybridClock initialized: now 1790469639709408 us; error 31 us; skew 500 ppm
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:40:39.713027 11851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:40:39.714414 11851 webserver.cc:460] Webserver started at http://127.0.98.2:33829/ using document root <none> and password file <none>
I20260927 00:40:39.715307 11851 fs_manager.cc:359] Metadata directory not provided
I20260927 00:40:39.715427 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:40:39.715723 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:39.738327 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "20dfd15e732d419d81a7873ccad9603b"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "2ab72e4eb8ef132bdafd7da93a5dfd1f"
server_key_iv: "58e5dbb1ac4110d50b15756dc3f36ad3"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.739246 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20dfd15e732d419d81a7873ccad9603b"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "2ab72e4eb8ef132bdafd7da93a5dfd1f"
server_key_iv: "58e5dbb1ac4110d50b15756dc3f36ad3"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.748070 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260927 00:40:39.769528 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:39.771863 11851 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260927 00:40:39.772070 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "20dfd15e732d419d81a7873ccad9603b"
format_stamp: "Formatted at 2026-09-27 00:40:39 on dist-test-slave-xltq"
server_key: "2ab72e4eb8ef132bdafd7da93a5dfd1f"
server_key_iv: "58e5dbb1ac4110d50b15756dc3f36ad3"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:39.772277 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:40:39.827768 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:40:39.831720 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:40:39.832005 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:40:39.834507 11851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:40:39.835750 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:40:39.835824 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:39.835886 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:40:39.835913 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:39.852382 11851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38207
I20260927 00:40:39.854146 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:40:39.856904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 11851
I20260927 00:40:39.857051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:40:39.857354 392 external_mini_cluster.cc:1428] Setting key 009d046492c53901f0d757831077d735
I20260927 00:40:39.859431 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38207 every 8 connection(s)
Sep 27 00:40:39 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:40:39.880285 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:39.863787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57443 (local address 127.0.98.62:40239)
0927 00:40:39.863936 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:39.863941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:39.864505 (+ 564us) server_negotiation.cc:407] Connection header received
0927 00:40:39.865990 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:39.865994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:39.866056 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:39.866115 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:40:39.868064 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.868971 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.870962 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.871320 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.871415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:39.874970 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:40:39.874975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:40:39.874978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:40:39.875017 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:40:39.877176 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:39.878049 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:39.878054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:39.878056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:39.878165 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:39.878621 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:39.878625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:39.878626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:39.878915 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:40:39.879012 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:39.879919 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 00:40:39.880080 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:40:39.882761 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:39.862533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.2:57443)
0927 00:40:39.864311 (+ 1778us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:39.864384 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:39.865682 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:39.866583 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:39.866594 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:39.867007 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:40:39.867842 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:39.867868 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.869173 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.869176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:39.870789 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:39.870803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:39.871652 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:39.871656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:39.871902 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:39.872733 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:40:39.872758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:40:39.874696 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:40:39.877391 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:39.877399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:39.877421 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:39.877890 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:39.878285 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:39.878288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:39.878291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:39.878501 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:39.879157 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:40:39.879164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:40:39.879570 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 00:40:39.880876 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0927 00:40:39.881543 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1471,"thread_start_us":201,"threads_started":1}
I20260927 00:40:39.884531 11981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260927 00:40:39.884881 11981 heartbeater.cc:464] Registering TS with master...
I20260927 00:40:39.885617 11981 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260927 00:40:39.887040 11631 ts_manager.cc:194] Registered new tserver with Master: 20dfd15e732d419d81a7873ccad9603b (127.0.98.2:38207)
I20260927 00:40:39.887948 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:40:39.917580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:40:40 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:40:40.105126 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:40:40.105557 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:40:40.105713 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:40:40.115195 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:40:40.115280 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:40:40.115365 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:40:40.123848 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:40:40.125463 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:40:40.127152 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:40:40.142968 11986 server_base.cc:1034] running on GCE node
W20260927 00:40:40.142884 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:40.143316 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:40:40.144057 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:40:40.144645 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:40:40.145589 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:40:40.153028 11986 hybrid_clock.cc:648] HybridClock initialized: now 1790469640152777 us; error 248 us; skew 500 ppm
Sep 27 00:40:40 dist-test-slave-xltq krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:40:40.157843 11986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:40:40.159161 11986 webserver.cc:460] Webserver started at http://127.0.98.3:37679/ using document root <none> and password file <none>
I20260927 00:40:40.159958 11986 fs_manager.cc:359] Metadata directory not provided
I20260927 00:40:40.160068 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:40:40.160365 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:40:40 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:40.195264 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "34d18ea7f49249a1a5264f58af80bfe2"
format_stamp: "Formatted at 2026-09-27 00:40:40 on dist-test-slave-xltq"
server_key: "528a2d0534b2d0ebfcb982a3156a2f36"
server_key_iv: "23334bf6a64884fbad80abf361631e70"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:40.197598 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34d18ea7f49249a1a5264f58af80bfe2"
format_stamp: "Formatted at 2026-09-27 00:40:40 on dist-test-slave-xltq"
server_key: "528a2d0534b2d0ebfcb982a3156a2f36"
server_key_iv: "23334bf6a64884fbad80abf361631e70"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:40.207297 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260927 00:40:40.231879 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:40.233758 11986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:40:40.234002 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "34d18ea7f49249a1a5264f58af80bfe2"
format_stamp: "Formatted at 2026-09-27 00:40:40 on dist-test-slave-xltq"
server_key: "528a2d0534b2d0ebfcb982a3156a2f36"
server_key_iv: "23334bf6a64884fbad80abf361631e70"
server_key_version: "kuduclusterkey@0"
I20260927 00:40:40.234226 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:40:40.260893 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:40:40.265013 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:40:40.265295 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:40:40.266073 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:40:40.267261 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:40:40.267380 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:40.267488 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:40:40.267556 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:40:40.286361 11986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42563
I20260927 00:40:40.287743 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:40:40.289958 12114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42563 every 8 connection(s)
I20260927 00:40:40.292827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 11986
I20260927 00:40:40.292953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:40:40.293256 392 external_mini_cluster.cc:1428] Setting key 78a0072f1e98fac1d693a8893f40051c
Sep 27 00:40:40 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:40:40.313800 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.292073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41453 (local address 127.0.98.62:40239)
0927 00:40:40.293781 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.293787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.296882 (+ 3095us) server_negotiation.cc:407] Connection header received
0927 00:40:40.298308 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.298312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.298375 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.298436 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:40:40.300496 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.301395 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.303586 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.303971 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.304057 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.308837 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:40:40.308841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:40:40.308844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:40:40.308875 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:40:40.310683 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:40.311392 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:40.311395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:40.311397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:40.311488 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:40.312081 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:40.312085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:40.312087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:40.312389 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:40:40.312490 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.313477 (+ 987us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.313627 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1584}
I20260927 00:40:40.315321 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.292618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40239 (local address 127.0.98.3:41453)
0927 00:40:40.296656 (+ 4038us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.296762 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.298018 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.298975 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.298987 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.299423 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:40:40.300297 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.300322 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.301627 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.301631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.303430 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.303443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.304324 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.304328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.304537 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.305348 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:40:40.305371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:40:40.308620 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:40:40.310869 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:40.310876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:40.310896 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:40.311279 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:40.311594 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:40:40.311597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:40:40.311599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:40:40.311935 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:40:40.312523 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:40:40.312531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:40:40.313104 (+ 573us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.313418 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.313860 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3726,"thread_start_us":182,"threads_started":1}
I20260927 00:40:40.316848 12115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40239
I20260927 00:40:40.317157 12115 heartbeater.cc:464] Registering TS with master...
I20260927 00:40:40.317837 12115 heartbeater.cc:511] Master 127.0.98.62:40239 requested a full tablet report, sending...
I20260927 00:40:40.319149 11631 ts_manager.cc:194] Registered new tserver with Master: 34d18ea7f49249a1a5264f58af80bfe2 (127.0.98.3:42563)
I20260927 00:40:40.319898 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41453
I20260927 00:40:40.324747 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:40:40.338445 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.327589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.62:40239)
0927 00:40:40.327721 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.327726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.328533 (+ 807us) server_negotiation.cc:407] Connection header received
0927 00:40:40.328744 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.328748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.328804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.328857 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:40:40.329859 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.330654 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.331901 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.332349 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.332437 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.334392 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:40:40.334397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:40:40.334399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:40:40.334436 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:40:40.336299 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:40.336921 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:40.336925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:40.336927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:40.337038 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:40:40.337357 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:40:40.337361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:40:40.337362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:40:40.337623 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:40:40.337740 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.338144 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.338273 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:40:40.342973 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:40:40.347728 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:40:40.375844 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.360667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42563 (local address 127.0.0.1:43490)
0927 00:40:40.364428 (+ 3761us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.365596 (+ 1168us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.365738 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.366881 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.366887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.366907 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:40:40.367068 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.367075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.369821 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.369824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.372480 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.372495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.374060 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.374064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.374203 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.374246 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.374365 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.374521 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3571,"thread_start_us":112,"threads_started":1}
I20260927 00:40:40.376886 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.361704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.0.98.3:42563)
0927 00:40:40.365510 (+ 3806us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.365523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.365899 (+ 376us) server_negotiation.cc:407] Connection header received
0927 00:40:40.365987 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.365993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.366246 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.366405 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:40:40.367229 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.369666 (+ 2437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.372674 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.373660 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.373784 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.373972 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.376589 (+ 2617us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.376732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3687,"thread_start_us":151,"threads_started":1}
I20260927 00:40:40.380581 12049 tablet_service.cc:1467] Processing CreateTablet for tablet b34568296e0d4d42ae238b45aeb2154f (DEFAULT_TABLE table=test-table [id=7894b0888ecb40a58abc5c7c489a4723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:40:40.382875 12049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34568296e0d4d42ae238b45aeb2154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:40:40.385367 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.363824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38207 (local address 127.0.0.1:33220)
0927 00:40:40.368262 (+ 4438us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.368277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.368360 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.381164 (+ 12804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.381168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.381196 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:40:40.381310 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.381317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.382706 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.382709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.383820 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.383833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.384951 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.384955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.385052 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.385084 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.385135 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.385216 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4352,"thread_start_us":104,"threads_started":1}
I20260927 00:40:40.386334 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.364719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.98.2:38207)
0927 00:40:40.380428 (+ 15709us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.380444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.380471 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:40:40.380565 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.380571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.380797 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.380979 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:40:40.381462 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.382574 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.384000 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.384811 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.385888 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.386047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.386101 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.386198 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15585,"thread_start_us":80,"threads_started":1}
I20260927 00:40:40.389554 11914 tablet_service.cc:1467] Processing CreateTablet for tablet b34568296e0d4d42ae238b45aeb2154f (DEFAULT_TABLE table=test-table [id=7894b0888ecb40a58abc5c7c489a4723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:40:40.390661 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34568296e0d4d42ae238b45aeb2154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:40:40.400580 12134 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Bootstrap starting.
I20260927 00:40:40.403045 12134 tablet_bootstrap.cc:654] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Neither blocks nor log segments found. Creating new log.
I20260927 00:40:40.413293 12135 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Bootstrap starting.
I20260927 00:40:40.413393 12134 log.cc:824] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:40:40.414116 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.363810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35247 (local address 127.0.0.1:46180)
0927 00:40:40.366667 (+ 2857us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.366682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.366768 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.405619 (+ 38851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.405623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.405645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:40:40.405765 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.405772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.409886 (+ 4114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.409890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.411209 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.411225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.413623 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.413627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.413739 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.413777 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.413840 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.413940 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2788,"thread_start_us":112,"threads_started":1}
I20260927 00:40:40.419478 12135 tablet_bootstrap.cc:654] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Neither blocks nor log segments found. Creating new log.
I20260927 00:40:40.419889 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.364304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.98.1:35247)
0927 00:40:40.405107 (+ 40803us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.405115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.405141 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:40:40.405222 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.405228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.405455 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.405922 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:40:40.406074 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.407262 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.411424 (+ 4162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.412637 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.415785 (+ 3148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.416030 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.416105 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.419726 (+ 3621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40662,"thread_start_us":77,"threads_started":1}
I20260927 00:40:40.421180 12135 log.cc:824] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:40:40.423696 12135 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: No bootstrap required, opened a new log
I20260927 00:40:40.423910 12135 ts_tablet_manager.cc:1397] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 00:40:40.423503 11765 tablet_service.cc:1467] Processing CreateTablet for tablet b34568296e0d4d42ae238b45aeb2154f (DEFAULT_TABLE table=test-table [id=7894b0888ecb40a58abc5c7c489a4723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:40:40.424739 11765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34568296e0d4d42ae238b45aeb2154f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:40:40.426035 12134 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: No bootstrap required, opened a new log
I20260927 00:40:40.426256 12134 ts_tablet_manager.cc:1397] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20260927 00:40:40.429870 12135 raft_consensus.cc:348] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.430490 12135 raft_consensus.cc:374] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:40:40.430591 12135 raft_consensus.cc:729] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d18ea7f49249a1a5264f58af80bfe2, State: Initialized, Role: FOLLOWER
I20260927 00:40:40.431210 12135 consensus_queue.cc:260] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.431180 12134 raft_consensus.cc:348] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.431447 12134 raft_consensus.cc:374] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:40:40.431527 12134 raft_consensus.cc:729] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20dfd15e732d419d81a7873ccad9603b, State: Initialized, Role: FOLLOWER
I20260927 00:40:40.432044 12134 consensus_queue.cc:260] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.432670 12135 ts_tablet_manager.cc:1428] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 00:40:40.433043 12134 ts_tablet_manager.cc:1428] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:40:40.433097 12115 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260927 00:40:40.433439 11981 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260927 00:40:40.440742 12139 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: Bootstrap starting.
I20260927 00:40:40.444466 12139 tablet_bootstrap.cc:654] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: Neither blocks nor log segments found. Creating new log.
I20260927 00:40:40.445551 12139 log.cc:824] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: Log is configured to *not* fsync() on all Append() calls
I20260927 00:40:40.446655 12140 raft_consensus.cc:482] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:40:40.446841 12140 raft_consensus.cc:504] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.448036 12140 leader_election.cc:288] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d18ea7f49249a1a5264f58af80bfe2 (127.0.98.3:42563), 0316eb8a129f466fbf40b0f62ab06f05 (127.0.98.1:35247)
I20260927 00:40:40.454747 12139 tablet_bootstrap.cc:492] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: No bootstrap required, opened a new log
I20260927 00:40:40.455184 12139 ts_tablet_manager.cc:1397] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260927 00:40:40.459697 12139 raft_consensus.cc:348] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.460251 12139 raft_consensus.cc:374] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:40:40.460361 12139 raft_consensus.cc:729] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0316eb8a129f466fbf40b0f62ab06f05, State: Initialized, Role: FOLLOWER
I20260927 00:40:40.461028 12139 consensus_queue.cc:260] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.462783 12139 ts_tablet_manager.cc:1428] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:40:40.463300 11843 heartbeater.cc:503] Master 127.0.98.62:40239 was elected leader, sending a full tablet report...
I20260927 00:40:40.477289 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.452477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34421 (local address 127.0.98.1:35247)
0927 00:40:40.452604 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.452609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.472540 (+ 19931us) server_negotiation.cc:407] Connection header received
0927 00:40:40.472756 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.472759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.472807 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.472850 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:40:40.473346 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.474114 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.475928 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.476562 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.476652 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.476726 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.477065 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.477151 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:40:40.477782 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.448327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35247 (local address 127.0.98.2:34421)
0927 00:40:40.472435 (+ 24108us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.472454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.472644 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.472996 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.472999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.473015 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:40:40.473213 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.473220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.474246 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.474250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.475767 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.475778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.476821 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.476825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.476925 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.476950 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.477530 (+ 580us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.477628 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24008,"mutex_wait_us":39,"spinlock_wait_cycles":22784,"thread_start_us":136,"threads_started":1}
I20260927 00:40:40.478430 11788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34568296e0d4d42ae238b45aeb2154f" candidate_uuid: "20dfd15e732d419d81a7873ccad9603b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0316eb8a129f466fbf40b0f62ab06f05" is_pre_election: true
I20260927 00:40:40.478799 11788 raft_consensus.cc:2454] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dfd15e732d419d81a7873ccad9603b in term 0.
I20260927 00:40:40.479339 11869 leader_election.cc:302] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0316eb8a129f466fbf40b0f62ab06f05, 20dfd15e732d419d81a7873ccad9603b; no voters:
I20260927 00:40:40.479682 12140 raft_consensus.cc:2781] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:40:40.479766 12140 raft_consensus.cc:482] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:40:40.479809 12140 raft_consensus.cc:3037] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:40:40.481899 12140 raft_consensus.cc:504] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.482352 12140 leader_election.cc:288] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [CANDIDATE]: Term 1 election: Requested vote from peers 34d18ea7f49249a1a5264f58af80bfe2 (127.0.98.3:42563), 0316eb8a129f466fbf40b0f62ab06f05 (127.0.98.1:35247)
I20260927 00:40:40.482914 11788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34568296e0d4d42ae238b45aeb2154f" candidate_uuid: "20dfd15e732d419d81a7873ccad9603b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0316eb8a129f466fbf40b0f62ab06f05"
I20260927 00:40:40.483055 11788 raft_consensus.cc:3037] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:40:40.484370 11788 raft_consensus.cc:2454] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dfd15e732d419d81a7873ccad9603b in term 1.
I20260927 00:40:40.484730 11869 leader_election.cc:302] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0316eb8a129f466fbf40b0f62ab06f05, 20dfd15e732d419d81a7873ccad9603b; no voters:
I20260927 00:40:40.485105 12140 raft_consensus.cc:2781] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:40:40.485347 12140 raft_consensus.cc:686] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 1 LEADER]: Becoming Leader. State: Replica: 20dfd15e732d419d81a7873ccad9603b, State: Running, Role: LEADER
I20260927 00:40:40.485743 12140 consensus_queue.cc:237] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } }
I20260927 00:40:40.490620 11631 catalog_manager.cc:5495] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b reported cstate change: term changed from 0 to 1, leader changed from <none> to 20dfd15e732d419d81a7873ccad9603b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "20dfd15e732d419d81a7873ccad9603b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20dfd15e732d419d81a7873ccad9603b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 } health_report { overall_health: UNKNOWN } } }
I20260927 00:40:40.498147 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.448399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44467 (local address 127.0.98.3:42563)
0927 00:40:40.448520 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.448524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.472601 (+ 24077us) server_negotiation.cc:407] Connection header received
0927 00:40:40.472801 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.472804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.472859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.472906 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:40:40.473331 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.474109 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.496635 (+ 22526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.497303 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.497397 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.497473 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.497881 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.497995 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:40:40.498943 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.448646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42563 (local address 127.0.98.2:44467)
0927 00:40:40.472488 (+ 23842us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.472504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:40:40.472681 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:40:40.473038 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:40:40.473040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:40:40.473070 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:40:40.473197 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.473204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.474276 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.474279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.495352 (+ 21073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:40:40.495372 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.497596 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.497600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:40:40.497725 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.497752 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:40:40.498589 (+ 837us) client_negotiation.cc:240] Negotiation successful
0927 00:40:40.498737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23762,"mutex_wait_us":48,"thread_start_us":94,"threads_started":1}
I20260927 00:40:40.499957 12060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34568296e0d4d42ae238b45aeb2154f" candidate_uuid: "20dfd15e732d419d81a7873ccad9603b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d18ea7f49249a1a5264f58af80bfe2" is_pre_election: true
I20260927 00:40:40.500399 12060 raft_consensus.cc:2454] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20dfd15e732d419d81a7873ccad9603b in term 0.
I20260927 00:40:40.500784 12065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34568296e0d4d42ae238b45aeb2154f" candidate_uuid: "20dfd15e732d419d81a7873ccad9603b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d18ea7f49249a1a5264f58af80bfe2"
I20260927 00:40:40.500922 12065 raft_consensus.cc:3037] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:40:40.502346 12065 raft_consensus.cc:2454] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20dfd15e732d419d81a7873ccad9603b in term 1.
I20260927 00:40:40.532523 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.527615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.0.98.2:38207)
0927 00:40:40.527749 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:40:40.527753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:40:40.527771 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:40:40.527873 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:40:40.527877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:40:40.527954 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:40:40.528070 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:40:40.528366 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.529145 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.530467 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:40:40.530823 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:40:40.530926 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:40:40.531382 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:40:40.531993 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:40:40.532127 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:40:40.532304 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:40:40.532373 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:40:40.541352 11788 raft_consensus.cc:1261] T b34568296e0d4d42ae238b45aeb2154f P 0316eb8a129f466fbf40b0f62ab06f05 [term 1 FOLLOWER]: Refusing update from remote peer 20dfd15e732d419d81a7873ccad9603b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:40:40.542340 12065 raft_consensus.cc:1261] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 1 FOLLOWER]: Refusing update from remote peer 20dfd15e732d419d81a7873ccad9603b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:40:40.542291 12145 consensus_queue.cc:1035] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0316eb8a129f466fbf40b0f62ab06f05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:40:40.542919 12116 tablet.cc:2369] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:40:40.549021 12145 consensus_queue.cc:1035] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d18ea7f49249a1a5264f58af80bfe2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 00:40:40 dist-test-slave-xltq krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:40:40.564270 12149 mvcc.cc:205] Tried to move back new op lower bound from 7333763647646765056 to 7333763647438733312. Current Snapshot: MvccSnapshot[applied={T|T < 7333763647646765056}]
I20260927 00:40:40.569664 12150 mvcc.cc:205] Tried to move back new op lower bound from 7333763647646765056 to 7333763647438733312. Current Snapshot: MvccSnapshot[applied={T|T < 7333763647646765056}]
I20260927 00:40:40.580170 11630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:40:40.580446 11630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:40:40.583333 11630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3a02b8cecb54856964b33093902eb43: Sending DeleteTablet for 3 replicas of tablet b34568296e0d4d42ae238b45aeb2154f
I20260927 00:40:40.584869 11914 tablet_service.cc:1514] Processing DeleteTablet for tablet b34568296e0d4d42ae238b45aeb2154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260927 00:40:40.585117 12049 tablet_service.cc:1514] Processing DeleteTablet for tablet b34568296e0d4d42ae238b45aeb2154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43490
I20260927 00:40:40.585790 11765 tablet_service.cc:1514] Processing DeleteTablet for tablet b34568296e0d4d42ae238b45aeb2154f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46180
I20260927 00:40:40.587343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 11712
I20260927 00:40:40.592433 12158 tablet_replica.cc:330] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: stopping tablet replica
I20260927 00:40:40.592835 12158 raft_consensus.cc:2229] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:40:40.593325 12158 raft_consensus.cc:2258] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:40:40.595062 12158 ts_tablet_manager.cc:1905] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:40:40.596645 12159 tablet_replica.cc:330] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: stopping tablet replica
I20260927 00:40:40.597003 12159 raft_consensus.cc:2229] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:40:40.597365 12159 raft_consensus.cc:2258] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:40:40.597913 11616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35247 (error 108)
I20260927 00:40:40.599927 12159 ts_tablet_manager.cc:1905] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:40:40.602751 12159 ts_tablet_manager.cc:1918] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:40:40.602873 12159 log.cc:1196] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/b34568296e0d4d42ae238b45aeb2154f
I20260927 00:40:40.603209 12159 ts_tablet_manager.cc:1939] T b34568296e0d4d42ae238b45aeb2154f P 34d18ea7f49249a1a5264f58af80bfe2: Deleting consensus metadata
I20260927 00:40:40.603191 12158 ts_tablet_manager.cc:1918] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:40:40.603286 12158 log.cc:1196] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/b34568296e0d4d42ae238b45aeb2154f
I20260927 00:40:40.603677 12158 ts_tablet_manager.cc:1939] T b34568296e0d4d42ae238b45aeb2154f P 20dfd15e732d419d81a7873ccad9603b: Deleting consensus metadata
I20260927 00:40:40.604696 11618 catalog_manager.cc:4841] TS 34d18ea7f49249a1a5264f58af80bfe2 (127.0.98.3:42563): tablet b34568296e0d4d42ae238b45aeb2154f (table test-table [id=7894b0888ecb40a58abc5c7c489a4723]) successfully deleted
I20260927 00:40:40.605453 11618 catalog_manager.cc:4841] TS 20dfd15e732d419d81a7873ccad9603b (127.0.98.2:38207): tablet b34568296e0d4d42ae238b45aeb2154f (table test-table [id=7894b0888ecb40a58abc5c7c489a4723]) successfully deleted
I20260927 00:40:40.605558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 11851
I20260927 00:40:40.609876 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:40:40.609566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35247 (local address 127.0.0.1:46196)
0927 00:40:40.609691 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:40:40.609776 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 00:40:40.610139 11616 catalog_manager.cc:4584] TS 0316eb8a129f466fbf40b0f62ab06f05 (127.0.98.1:35247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b34568296e0d4d42ae238b45aeb2154f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35247: connect: Connection refused (error 111)
I20260927 00:40:40.614197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 11986
I20260927 00:40:40.620311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 11599
I20260927 00:40:40.628612 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:40:40.975554 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:40:40.975742 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:40:41.322975 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:40:41.323 UTC [11185] LOG: received smart shutdown request
2026-09-27 00:40:41.327 UTC [11185] LOG: background worker "logical replication launcher" (PID 11194) exited with exit code 1
2026-09-27 00:40:41.327 UTC [11189] LOG: shutting down
2026-09-27 00:40:41.554 UTC [11185] LOG: database system is shut down
2026-09-27T00:40:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49745 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:40:41 dist-test-slave-xltq krb5kdc[12171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:40:41 dist-test-slave-xltq krb5kdc[12171](info): set up 2 sockets
Sep 27 00:40:41 dist-test-slave-xltq krb5kdc[12171](info): commencing operation
krb5kdc: starting...
W20260927 00:40:42.636183 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:40:42 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:40:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:40:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:40:42.815995 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSbnIrP/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:40:45.611 UTC [12201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:40:45.611 UTC [12201] LOG: listening on IPv4 address "127.0.98.20", port 52265
2026-09-27 00:40:45.615 UTC [12201] LOG: listening on Unix socket "/tmp/.s.PGSQL.52265"
2026-09-27 00:40:45.639 UTC [12204] LOG: database system was shut down at 2026-09-27 00:40:43 UTC
2026-09-27 00:40:45.643 UTC [12201] LOG: database system is ready to accept connections
I20260927 00:40:46.520103 392 mini_postgres.cc:96] Postgres bound to 52265
2026-09-27 00:40:46.526 UTC [12212] FATAL: database "slave" does not exist
127.0.98.20:52265 - accepting connections
I20260927 00:40:46.527245 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:40:46.542865 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:40:47.113026 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:40:47.113097 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin
2026-09-27 00:40:47,386 [I] DB FLAVOR :POSTGRES
2026-09-27 00:40:47,387 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:40:47,387 [I] Checking connection..
2026-09-27 00:40:47,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:40:47,631 [I] Checking connection passed.
2026-09-27 00:40:47,632 [I] --------- Verifying version history table ---------
2026-09-27 00:40:47,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:40:47,886 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:40:47,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:40:48,148 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:40:48,148 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:40:48,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:40:48.419 UTC [12276] WARNING: there is no transaction in progress
2026-09-27 00:40:48,424 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:40:48,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:40:48,692 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:40:48,692 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:40:48,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:40:48,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:40:49,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xltq.c.gcp-upstream.internal','N') ;"
2026-09-27 00:40:49,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:40:49,704 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:40:49,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:40:49,947 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:40:49,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:40:50,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:40:50,446 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:40:50,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:40:51.536 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.554 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.573 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.590 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.607 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.629 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.699 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.707 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.724 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.735 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.748 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.764 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.776 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.784 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.792 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:51.799 UTC [12384] WARNING: there is no transaction in progress
2026-09-27 00:40:52,573 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:40:52,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:40:52,835 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:40:52,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:40:53,106 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:40:53,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:40:53,352 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:40:53,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:40:53,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:40:53,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xltq.c.gcp-upstream.internal';"
2026-09-27 00:40:54,077 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:40:54,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:40:54,322 [I] DB_PATCHES have already been applied
I20260927 00:40:54.328162 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSbnIrP/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:40:54.328320 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:40:54.332512 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46469
Sep 27, 2026 12:40:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:40:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:40:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:40:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:40:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:40:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:40:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:40:55.368788 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.842s user 0.001s sys 0.010s
Sep 27, 2026 12:41:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:41:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:41:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:41:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:41:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:41:18.694792 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:41:18 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469678, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:41:18.788487 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:41:18.798730 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:41:19.724965 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:41:19.725051 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms
I20260927 00:41:19.725070 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52265
2026-09-27 00:41:19,893 [I] DB FLAVOR :POSTGRES
2026-09-27 00:41:19,894 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:41:19,894 [I] Checking connection
2026-09-27 00:41:19,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:41:20,158 [I] connection success
2026-09-27 00:41:20,158 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:41:20,158 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:41:20,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:41:20,399 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:41:20,400 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:41:20,400 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:41:20,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:41:20,678 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:41:21.206264 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:41:21.429783 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:41:21.480690 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:41:21 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469678, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:21.923907 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:41:21.924078 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSbnIrP/build/debug/bin/postgresql.jar:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSbnIrP/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:41:21.924103 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:41:21.927927 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12565
I20260927 00:41:21.928022 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52767
Sep 27, 2026 12:41:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:41:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:41:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:41:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:41:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:41:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:41:24.006318 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.218s user 0.007s sys 0.144s
I20260927 00:41:24.006423 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:41:24.006494 392 mini_ranger_kms.cc:401] 127.0.98.20:52767/kms/v1/keys
Sep 27 00:41:24 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:41:24 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:41:24 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:41:27 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:27.331982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44059
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44059
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:27.445788 12615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:27.446077 12615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:27.446120 12615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:27.450285 12615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:27.450347 12615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:27.450366 12615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:27.450382 12615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:27.450395 12615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:27.456095 12615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44059
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:27.457254 12615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:27.458315 12615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:41:27.467948 12615 server_base.cc:1034] running on GCE node
W20260927 00:41:27.467895 12620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.467954 12621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.467890 12623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.469012 12615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:27.470239 12615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:27.471416 12615 hybrid_clock.cc:648] HybridClock initialized: now 1790469687471428 us; error 74 us; skew 500 ppm
Sep 27 00:41:27 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:27.474361 12615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:27.475301 12615 webserver.cc:460] Webserver started at http://127.0.98.62:46171/ using document root <none> and password file <none>
I20260927 00:41:27.475814 12615 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:27.475867 12615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:27.476037 12615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:41:27 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:27.529395 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "9d38c262c6964eb4a8bd5c88a236a574"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-xltq"
server_key: "cb3f647643135c816bd05d0e3b2aa9f3"
server_key_iv: "624ba725a6019744fc07ccfcdbb62cef"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:27.530258 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d38c262c6964eb4a8bd5c88a236a574"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-xltq"
server_key: "cb3f647643135c816bd05d0e3b2aa9f3"
server_key_iv: "624ba725a6019744fc07ccfcdbb62cef"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:27.535920 12615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 00:41:27.554731 12630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.556599 12615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:27.556779 12615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "9d38c262c6964eb4a8bd5c88a236a574"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-xltq"
server_key: "cb3f647643135c816bd05d0e3b2aa9f3"
server_key_iv: "624ba725a6019744fc07ccfcdbb62cef"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:27.556938 12615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:27.577111 12615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:27.580955 12615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:27.581178 12615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:27.589665 12615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44059
I20260927 00:41:27.589776 12682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44059 every 8 connection(s)
I20260927 00:41:27.590898 12615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:27.593982 12683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.599931 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Bootstrap starting.
I20260927 00:41:27.600045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 12615
I20260927 00:41:27.600152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:27.600514 392 external_mini_cluster.cc:1428] Setting key e1154e5c693976ab41fa7724110083d9
I20260927 00:41:27.602803 12683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.603570 12683 log.cc:824] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:27.605826 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: No bootstrap required, opened a new log
I20260927 00:41:27.609941 12683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:27.610251 12683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.610340 12683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d38c262c6964eb4a8bd5c88a236a574, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.610896 12683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:27.611043 12683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:27.611122 12683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:27.611220 12683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:41:27 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.612277 12683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:27.612660 12683 leader_election.cc:302] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d38c262c6964eb4a8bd5c88a236a574; no voters:
I20260927 00:41:27.613013 12683 leader_election.cc:288] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:27.613112 12688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:27.613319 12688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 LEADER]: Becoming Leader. State: Replica: 9d38c262c6964eb4a8bd5c88a236a574, State: Running, Role: LEADER
I20260927 00:41:27.613710 12688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:27.614542 12683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:27.614988 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d38c262c6964eb4a8bd5c88a236a574. Latest consensus state: current_term: 1 leader_uuid: "9d38c262c6964eb4a8bd5c88a236a574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } } }
I20260927 00:41:27.615129 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:27.615338 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d38c262c6964eb4a8bd5c88a236a574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } } }
I20260927 00:41:27.615459 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:27.616916 12683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:41:27.617861 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.601950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.98.62:44059)
0927 00:41:27.602447 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.602466 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.602512 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:41:27.603571 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.603599 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.604070 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.604517 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.605376 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.606924 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.608237 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.608962 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.609122 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.612111 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.612118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.612155 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.612223 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.614335 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.614922 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.614928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.614935 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.615081 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.615403 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.615406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.615407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.615961 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.616137 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.616579 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.616876 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260927 00:41:27.623088 12692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:27.626598 12692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:27.634570 12692 catalog_manager.cc:1347] Generated new cluster ID: 0266e7fe28db4b4f8cb96dccc66890cd
I20260927 00:41:27.634748 12692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:27.657830 12692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:27.659392 12692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:27.681228 12692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Generated new TSK 0
I20260927 00:41:27.682313 12692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:28.372857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:28.496456 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:28.496747 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:28.496793 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:28.501369 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:28.501446 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:28.501513 12728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:28.507310 12728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:28.508574 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:28.509680 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:28.520738 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:28.520736 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:28.521294 12728 server_base.cc:1034] running on GCE node
W20260927 00:41:28.521498 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:28.521998 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:28.522763 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:28.526197 12728 hybrid_clock.cc:648] HybridClock initialized: now 1790469688526152 us; error 56 us; skew 500 ppm
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:28.529804 12728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:28.530927 12728 webserver.cc:460] Webserver started at http://127.0.98.1:40847/ using document root <none> and password file <none>
I20260927 00:41:28.531646 12728 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:28.531720 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:28.531944 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:28.552003 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "df111d9c5c54442f97789140a4ae5bc9"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "591a684f9a78e1ddc5f99090fa5ae599"
server_key_iv: "51d60081754f95027d43a2f8ca64c137"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.553009 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df111d9c5c54442f97789140a4ae5bc9"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "591a684f9a78e1ddc5f99090fa5ae599"
server_key_iv: "51d60081754f95027d43a2f8ca64c137"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.562776 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260927 00:41:28.584605 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:28.586316 12728 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 00:41:28.586483 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "df111d9c5c54442f97789140a4ae5bc9"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "591a684f9a78e1ddc5f99090fa5ae599"
server_key_iv: "51d60081754f95027d43a2f8ca64c137"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.586623 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:28.604465 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:28.608397 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:28.608662 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:28.609418 12728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:28.610847 12728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:28.610925 12728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:28.611090 12728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:28.611171 12728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 00:41:28.622826 12728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260927 00:41:28.622865 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260927 00:41:28.624106 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:28.627142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 12728
I20260927 00:41:28.627269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:28.627609 392 external_mini_cluster.cc:1428] Setting key 73304265b052cbf7efd3babad070cfb3
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:28.649899 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:28.626797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58885 (local address 127.0.98.62:44059)
0927 00:41:28.628498 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:28.628507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:28.629039 (+ 532us) server_negotiation.cc:407] Connection header received
0927 00:41:28.631619 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:28.631624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:28.631690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:28.631793 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:28.633544 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:28.634413 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:28.636446 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:28.636821 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:28.636911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:28.644785 (+ 7874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:28.644792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:28.644796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:28.644838 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:28.646915 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:28.647763 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:28.647768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:28.647770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:28.647876 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:28.648409 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:28.648413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:28.648415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:28.648661 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:28.648749 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:28.649551 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 00:41:28.649702 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":112,"threads_started":1}
I20260927 00:41:28.651947 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:28.627220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.1:58885)
0927 00:41:28.628774 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:28.628823 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:28.629890 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:28.632107 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:28.632120 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:28.632543 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:28.633336 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:28.633362 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:28.634583 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:28.634587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:28.636256 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:28.636272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:28.637186 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:28.637190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:28.637439 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:28.638189 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:28.638212 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:28.642425 (+ 4213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:28.647141 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:28.647149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:28.647173 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:28.647616 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:28.648022 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:28.648025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:28.648027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:28.648287 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:28.648821 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:28.648828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:28.649241 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 00:41:28.650354 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0927 00:41:28.650764 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":165,"threads_started":1}
I20260927 00:41:28.653762 12859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:28.654021 12859 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:28.654559 12859 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:28.656219 12647 ts_manager.cc:194] Registered new tserver with Master: df111d9c5c54442f97789140a4ae5bc9 (127.0.98.1:45115)
I20260927 00:41:28.657388 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:28.676275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:41:28.844895 12867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:28.845317 12867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:28.845386 12867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:28.851702 12867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:28.851809 12867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:28.851895 12867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:28.860431 12867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:28.861963 12867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:28.863469 12867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:28.876711 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:28.876947 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:28.876765 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:28.877671 12867 server_base.cc:1034] running on GCE node
I20260927 00:41:28.878218 12867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:28.879105 12867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:28.880290 12867 hybrid_clock.cc:648] HybridClock initialized: now 1790469688880255 us; error 54 us; skew 500 ppm
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:28.884127 12867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:28.885438 12867 webserver.cc:460] Webserver started at http://127.0.98.2:38395/ using document root <none> and password file <none>
I20260927 00:41:28.886235 12867 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:28.886336 12867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:28.886639 12867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:41:28 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:28.910482 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "d596a6216c994323adc15ad0ccebcf12"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "5e440bef7d8f1ce12c4d1ec8c9de590a"
server_key_iv: "d9213e02d65eb2fe1c584d30f34e1e4f"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.911851 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d596a6216c994323adc15ad0ccebcf12"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "5e440bef7d8f1ce12c4d1ec8c9de590a"
server_key_iv: "d9213e02d65eb2fe1c584d30f34e1e4f"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.917667 12867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:41:28.950714 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:28.952673 12867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:41:28.952844 12867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "d596a6216c994323adc15ad0ccebcf12"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "5e440bef7d8f1ce12c4d1ec8c9de590a"
server_key_iv: "d9213e02d65eb2fe1c584d30f34e1e4f"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:28.953003 12867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:28.983180 12867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:28.987283 12867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:28.987548 12867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:28.988345 12867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:28.989508 12867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:28.989624 12867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:28.989732 12867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:28.989804 12867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:29.004487 12867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33135
I20260927 00:41:29.005964 12867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:29.016799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 12867
I20260927 00:41:29.016932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:29.017259 392 external_mini_cluster.cc:1428] Setting key 746e21c557a536cb066734e2e3f47320
I20260927 00:41:29.007462 12996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33135 every 8 connection(s)
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:29.030381 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:29.011295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51245 (local address 127.0.98.62:44059)
0927 00:41:29.011440 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:29.011445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:29.011945 (+ 500us) server_negotiation.cc:407] Connection header received
0927 00:41:29.013358 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:29.013363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:29.013432 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:29.013483 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:29.015413 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.016333 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.020397 (+ 4064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.020732 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.020844 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:29.025529 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:29.025534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:29.025537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:29.025576 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:29.027458 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.028301 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.028305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.028307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.028413 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.028907 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.028911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.028913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.029165 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:29.029244 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:29.030041 (+ 797us) server_negotiation.cc:299] Negotiation successful
0927 00:41:29.030190 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:41:29.032487 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:29.010107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.2:51245)
0927 00:41:29.011751 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:29.011822 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:29.013059 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:29.013957 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:29.013969 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:29.014372 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:29.015198 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:29.015219 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.018576 (+ 3357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.018580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:29.020164 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:29.020186 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.021100 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.021104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:29.021336 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:29.023255 (+ 1919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:29.023281 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:29.025214 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:29.027658 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:29.027665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:29.027686 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:29.028128 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:29.028534 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:29.028537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:29.028539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:29.028782 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:29.029319 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:29.029326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:29.029724 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 00:41:29.030876 (+ 1152us) client_negotiation.cc:240] Negotiation successful
0927 00:41:29.031268 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":160,"threads_started":1}
I20260927 00:41:29.034220 12997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:29.034473 12997 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:29.034996 12997 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:29.036105 12647 ts_manager.cc:194] Registered new tserver with Master: d596a6216c994323adc15ad0ccebcf12 (127.0.98.2:33135)
I20260927 00:41:29.036696 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51245
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:29.071640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:29.244863 13002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:29.245735 13002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:29.245872 13002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:29.252264 13002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:29.252480 13002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:29.252614 13002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:29.260970 13002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:29.262668 13002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:29.264104 13002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:29.279583 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:29.280221 13010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:29.280733 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:29.281738 13002 server_base.cc:1034] running on GCE node
I20260927 00:41:29.282426 13002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:29.283317 13002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:29.286608 13002 hybrid_clock.cc:648] HybridClock initialized: now 1790469689286537 us; error 108 us; skew 500 ppm
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:29.290618 13002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:29.291898 13002 webserver.cc:460] Webserver started at http://127.0.98.3:45173/ using document root <none> and password file <none>
I20260927 00:41:29.292722 13002 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:29.292879 13002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:29.293166 13002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:29.318976 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "755050658c04498d8cca4927aa8b3103"
format_stamp: "Formatted at 2026-09-27 00:41:29 on dist-test-slave-xltq"
server_key: "e644104e7a79501bbbee53f8b87abbdc"
server_key_iv: "cfb8cd6b8f8cdfb61edb19337f2a6803"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:29.320122 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "755050658c04498d8cca4927aa8b3103"
format_stamp: "Formatted at 2026-09-27 00:41:29 on dist-test-slave-xltq"
server_key: "e644104e7a79501bbbee53f8b87abbdc"
server_key_iv: "cfb8cd6b8f8cdfb61edb19337f2a6803"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:29.324973 13002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:41:29.348973 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:29.350956 13002 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260927 00:41:29.351229 13002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "755050658c04498d8cca4927aa8b3103"
format_stamp: "Formatted at 2026-09-27 00:41:29 on dist-test-slave-xltq"
server_key: "e644104e7a79501bbbee53f8b87abbdc"
server_key_iv: "cfb8cd6b8f8cdfb61edb19337f2a6803"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:29.351485 13002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:29.395152 13002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:29.403523 13002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:29.403825 13002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:29.404846 13002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:29.406018 13002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:29.408515 13002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260927 00:41:29.408663 13002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:29.408733 13002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:29.436944 13002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42475
I20260927 00:41:29.438272 13002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:29.444165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13002
I20260927 00:41:29.444310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:29.444630 392 external_mini_cluster.cc:1428] Setting key cc6e3a6450537a3191c479d2925091f6
I20260927 00:41:29.452939 13130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42475 every 8 connection(s)
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:29.492854 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:29.456407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36725 (local address 127.0.98.62:44059)
0927 00:41:29.456533 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:29.456537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:29.467978 (+ 11441us) server_negotiation.cc:407] Connection header received
0927 00:41:29.469265 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:29.469269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:29.469330 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:29.469432 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:29.471090 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.471943 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.473785 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.474140 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.474222 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:29.478485 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:29.478489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:29.478492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:29.478526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:29.480308 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.481008 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.481012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.481014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.481109 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.481553 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.481557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.481558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.481803 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:29.481882 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:29.492475 (+ 10593us) server_negotiation.cc:299] Negotiation successful
0927 00:41:29.492671 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:41:29.494974 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:29.455519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.3:36725)
0927 00:41:29.467752 (+ 12233us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:29.467840 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:29.469006 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:29.469788 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:29.469798 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:29.470187 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:29.470918 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:29.470939 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.472118 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.472121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:29.473637 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:29.473650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.474443 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.474447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:29.474686 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:29.476381 (+ 1695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:29.476405 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:29.478239 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:29.480477 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:29.480484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:29.480504 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:29.480882 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:29.481223 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:29.481226 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:29.481228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:29.481429 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:29.481950 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:29.481958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:29.482301 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 00:41:29.493357 (+ 11056us) client_negotiation.cc:240] Negotiation successful
0927 00:41:29.493815 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11898,"thread_start_us":168,"threads_started":1}
I20260927 00:41:29.496685 13131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:29.496953 13131 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:29.497674 13131 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:29.499046 12647 ts_manager.cc:194] Registered new tserver with Master: 755050658c04498d8cca4927aa8b3103 (127.0.98.3:42475)
I20260927 00:41:29.499861 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36725
I20260927 00:41:29.505904 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:29.506850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 12728
I20260927 00:41:29.515947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 12867
I20260927 00:41:29.524173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13002
I20260927 00:41:29.531185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 12615
I20260927 00:41:29.539225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44059
--webserver_interface=127.0.98.62
--webserver_port=46171
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44059
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:29.669821 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:29.670128 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:29.670167 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:29.674269 13134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:29.674358 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:29.674377 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:29.674392 13134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:29.674407 13134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:29.680233 13134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44059
--ranger_config_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:29.681337 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:29.682389 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:29.691777 13139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:29.691792 13140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:29.691859 13134 server_base.cc:1034] running on GCE node
W20260927 00:41:29.691792 13142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:29.692833 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:29.694114 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:29.695312 13134 hybrid_clock.cc:648] HybridClock initialized: now 1790469689695285 us; error 46 us; skew 500 ppm
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:29.698760 13134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:29.699878 13134 webserver.cc:460] Webserver started at http://127.0.98.62:46171/ using document root <none> and password file <none>
I20260927 00:41:29.700556 13134 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:29.700645 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:29.752802 13134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:29.755155 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:29.756713 13134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:29.756841 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "9d38c262c6964eb4a8bd5c88a236a574"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-xltq"
server_key: "cb3f647643135c816bd05d0e3b2aa9f3"
server_key_iv: "624ba725a6019744fc07ccfcdbb62cef"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:29.757493 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:29.774330 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:29.777941 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:29.778167 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:29.785964 13134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44059
I20260927 00:41:29.786000 13202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44059 every 8 connection(s)
I20260927 00:41:29.787269 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:29.788697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13134
I20260927 00:41:29.789093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45115
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44059
--webserver_port=40847
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:29.792461 13203 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:41:29.794214 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Bootstrap starting.
I20260927 00:41:29.810426 13203 log.cc:824] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:29.817210 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:29.817586 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Bootstrap complete.
I20260927 00:41:29.821746 13203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:29.822178 13203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d38c262c6964eb4a8bd5c88a236a574, State: Initialized, Role: FOLLOWER
I20260927 00:41:29.822700 13203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:29.822827 13203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:29.822896 13203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:29.822991 13203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:41:29.825044 13203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:29.825412 13203 leader_election.cc:302] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d38c262c6964eb4a8bd5c88a236a574; no voters:
I20260927 00:41:29.825738 13203 leader_election.cc:288] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:41:29.825861 13207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:41:29.826068 13207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [term 2 LEADER]: Becoming Leader. State: Replica: 9d38c262c6964eb4a8bd5c88a236a574, State: Running, Role: LEADER
I20260927 00:41:29.826434 13207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } }
I20260927 00:41:29.827004 13203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:29.827756 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d38c262c6964eb4a8bd5c88a236a574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } } }
I20260927 00:41:29.827893 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:29.828804 13216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:29.828990 13209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d38c262c6964eb4a8bd5c88a236a574. Latest consensus state: current_term: 2 leader_uuid: "9d38c262c6964eb4a8bd5c88a236a574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d38c262c6964eb4a8bd5c88a236a574" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44059 } } }
I20260927 00:41:29.829083 13209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:29.829295 13203 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:41:29.835489 13216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:29.836567 13216 catalog_manager.cc:1259] Loaded cluster ID: 0266e7fe28db4b4f8cb96dccc66890cd
I20260927 00:41:29.836679 13216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:29.839268 13216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:29.840086 13216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Loaded TSK: 0
I20260927 00:41:29.841466 13216 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:41:29.931239 13205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:29.931555 13205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:29.931614 13205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:29.937903 13205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:29.938084 13205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:29.938206 13205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:29.945434 13205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40847
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:29.946609 13205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:29.947721 13205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:29.959227 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:29.959264 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:29.959690 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:29.959970 13205 server_base.cc:1034] running on GCE node
I20260927 00:41:29.960485 13205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:29.961190 13205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:29.962421 13205 hybrid_clock.cc:648] HybridClock initialized: now 1790469689962404 us; error 56 us; skew 500 ppm
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:29.965435 13205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:29.966568 13205 webserver.cc:460] Webserver started at http://127.0.98.1:40847/ using document root <none> and password file <none>
I20260927 00:41:29.967316 13205 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:29.967437 13205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:41:29 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:29.989750 13205 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260927 00:41:29.997169 13251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:29.999142 13205 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260927 00:41:29.999387 13205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "df111d9c5c54442f97789140a4ae5bc9"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "591a684f9a78e1ddc5f99090fa5ae599"
server_key_iv: "51d60081754f95027d43a2f8ca64c137"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:30.000317 13205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:30.039392 13205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:30.043443 13205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:30.043668 13205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:30.044449 13205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:30.045661 13205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:30.045729 13205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.045789 13205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:30.045923 13205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 00:41:30.059509 13205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45115
I20260927 00:41:30.061313 13205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:30.063522 13364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45115 every 8 connection(s)
I20260927 00:41:30.071048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13205
I20260927 00:41:30.071538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33135
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44059
--webserver_port=38395
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:30.087865 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.064545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.1:36913)
0927 00:41:30.065818 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.065879 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.066982 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.070001 (+ 3019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.070009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.070298 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:30.070893 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.070916 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.073113 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.073118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.074508 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.074522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.075613 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.075618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.075844 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.076574 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:30.076595 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:30.078101 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:30.081600 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.081609 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.081628 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.082020 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.083703 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.083706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.083708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.083929 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.085823 (+ 1894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:30.085832 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:30.086207 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.086415 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.086788 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":147,"threads_started":1}
I20260927 00:41:30.094138 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.064340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36913 (local address 127.0.98.62:44059)
0927 00:41:30.068509 (+ 4169us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.068531 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.068581 (+ 50us) server_negotiation.cc:407] Connection header received
0927 00:41:30.068980 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.068998 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.069462 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.071164 (+ 1702us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:30.071480 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.072936 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.074672 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.075412 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.075592 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.078396 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:30.078404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:30.078423 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:30.078459 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:30.080607 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.083362 (+ 2755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.083371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.083378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.083571 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.084061 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.084064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.084066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.085664 (+ 1598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:30.089247 (+ 3583us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.089469 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.089820 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3986,"thread_start_us":131,"threads_started":1}
I20260927 00:41:30.099325 13365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:30.099646 13365 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.100404 13365 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.103421 13365 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:41:30.103596 13365 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.103832 13365 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:30.105217 13365 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.105456 13365 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.106652 13365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:41:30.213603 13369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:30.213914 13369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:30.213979 13369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:30.218459 13369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:30.218549 13369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:30.218762 13369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:30.225550 13369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38395
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:30.227325 13369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:30.228804 13369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:30.243680 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:30.243637 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:30.243965 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:30.244145 13369 server_base.cc:1034] running on GCE node
I20260927 00:41:30.244724 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:30.245675 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:30.246903 13369 hybrid_clock.cc:648] HybridClock initialized: now 1790469690246882 us; error 34 us; skew 500 ppm
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:30.250985 13369 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:30.252069 13369 webserver.cc:460] Webserver started at http://127.0.98.2:38395/ using document root <none> and password file <none>
I20260927 00:41:30.252698 13369 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:30.252785 13369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:30.278932 13369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:41:30.283115 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.285032 13369 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 00:41:30.285202 13369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "d596a6216c994323adc15ad0ccebcf12"
format_stamp: "Formatted at 2026-09-27 00:41:28 on dist-test-slave-xltq"
server_key: "5e440bef7d8f1ce12c4d1ec8c9de590a"
server_key_iv: "d9213e02d65eb2fe1c584d30f34e1e4f"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:30.286023 13369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:30.306012 13369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:30.309460 13369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:30.309700 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:30.310371 13369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:30.311414 13369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:30.311525 13369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.311630 13369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:30.311697 13369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.322224 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33135 every 8 connection(s)
I20260927 00:41:30.322194 13369 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33135
I20260927 00:41:30.323599 13369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:30.324325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13369
I20260927 00:41:30.324790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52767/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42475
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44059
--webserver_port=45173
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:30.340166 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.325603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59331 (local address 127.0.98.62:44059)
0927 00:41:30.327299 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.327304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.327946 (+ 642us) server_negotiation.cc:407] Connection header received
0927 00:41:30.329587 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.329591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.329652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.329726 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:30.330877 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.331412 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.332638 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.332879 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.332969 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.335550 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:30.335554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:30.335557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:30.335587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:30.337174 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.337967 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.337971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.337973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.338076 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.338558 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.338562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.338564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.338840 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:30.338920 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.339813 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.339978 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566}
I20260927 00:41:30.342185 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.326128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.2:59331)
0927 00:41:30.327716 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.327778 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.329349 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.329917 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.329925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.330207 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:30.330719 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.330732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.331542 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.331545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.332518 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.332528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.333026 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.333030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.333193 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.333769 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:30.333786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:30.335357 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:30.337355 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.337363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.337384 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.337833 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.338200 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.338203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.338204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.338437 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.339013 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:30.339021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:30.339466 (+ 445us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.340471 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.340919 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1358,"thread_start_us":159,"threads_started":1}
I20260927 00:41:30.343991 13499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:30.344390 13499 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.345207 13499 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.347373 13499 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:41:30.347575 13499 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.347820 13499 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:30.348765 13499 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.349010 13499 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.349792 13499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:41:30.477160 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:30.477464 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:30.477524 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:30.482096 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:30.482297 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:30.482440 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:30.490865 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52767/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45173
--tserver_master_addrs=127.0.98.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:30.492404 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:30.493821 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:30.508510 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:30.508510 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:30.509044 13501 server_base.cc:1034] running on GCE node
W20260927 00:41:30.512109 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:30.512830 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:30.513772 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:30.514997 13501 hybrid_clock.cc:648] HybridClock initialized: now 1790469690514979 us; error 29 us; skew 500 ppm
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:30.518811 13501 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:30.520087 13501 webserver.cc:460] Webserver started at http://127.0.98.3:45173/ using document root <none> and password file <none>
I20260927 00:41:30.520947 13501 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:30.521066 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:30.543546 13501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:30.546404 13519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.548161 13501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:30.548421 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "755050658c04498d8cca4927aa8b3103"
format_stamp: "Formatted at 2026-09-27 00:41:29 on dist-test-slave-xltq"
server_key: "e644104e7a79501bbbee53f8b87abbdc"
server_key_iv: "cfb8cd6b8f8cdfb61edb19337f2a6803"
server_key_version: "kuduclusterkey@0"
I20260927 00:41:30.549203 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:30.572326 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:30.576401 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:30.576723 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:30.577618 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:30.578902 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:30.579041 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.579164 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:30.579236 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:30.592474 13501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42475
I20260927 00:41:30.593765 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:30.594314 13632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42475 every 8 connection(s)
I20260927 00:41:30.597468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13501
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:30.609059 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.598196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.0.98.62:44059)
0927 00:41:30.598474 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.598479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.598744 (+ 265us) server_negotiation.cc:407] Connection header received
0927 00:41:30.598949 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.598953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.599006 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.599078 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:30.600178 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.600791 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.601960 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.602310 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.602427 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.604229 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:30.604234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:30.604237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:30.604264 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:30.607008 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.607533 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.607536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.607538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.607677 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.608015 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.608019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.608021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.608300 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:30.608498 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.608682 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.608872 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"spinlock_wait_cycles":9472,"thread_start_us":97,"threads_started":1}
I20260927 00:41:30.610287 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.595807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53557 (local address 127.0.98.62:44059)
0927 00:41:30.595976 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.595981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.596965 (+ 984us) server_negotiation.cc:407] Connection header received
0927 00:41:30.598173 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.598178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.598235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.598295 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:30.600148 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.601018 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.602516 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.602828 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.602908 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.605940 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:30.605945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:30.605947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:30.605996 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:30.607592 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.608186 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.608190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.608192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.608299 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:30.608694 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:30.608697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:30.608699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:30.608923 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:30.608996 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.610008 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.610127 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":1408}
I20260927 00:41:30.611364 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.596253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44059 (local address 127.0.98.3:53557)
0927 00:41:30.596773 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.596830 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.597898 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.598775 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.598786 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.599174 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:30.599894 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.599916 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.601186 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.601190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.602365 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.602378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.603153 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.603158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.603379 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.604018 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:30.604042 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:30.605705 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:30.607688 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.607695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.607714 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.608099 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.608378 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:30.608382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:30.608384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:30.608589 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:30.609080 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:30.609087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:30.609697 (+ 610us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.609955 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.610329 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":153,"threads_started":1}
I20260927 00:41:30.612779 13633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44059
I20260927 00:41:30.613152 13633 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.613716 13633 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.615556 13633 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44059 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:41:30.615710 13633 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.615949 13633 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:30.616967 13633 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:30.617162 13633 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
W20260927 00:41:30.618077 13633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:41:30 dist-test-slave-xltq krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:31.106997 13365 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:31.107314 13365 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:31.108340 13161 ts_manager.cc:194] Registered new tserver with Master: df111d9c5c54442f97789140a4ae5bc9 (127.0.98.1:45115)
I20260927 00:41:31.109755 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36913
I20260927 00:41:31.350100 13499 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:31.350438 13499 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:31.351305 13161 ts_manager.cc:194] Registered new tserver with Master: d596a6216c994323adc15ad0ccebcf12 (127.0.98.2:33135)
I20260927 00:41:31.351926 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59331
Sep 27 00:41:31 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:41:31.618356 13633 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:31.618697 13633 heartbeater.cc:511] Master 127.0.98.62:44059 requested a full tablet report, sending...
I20260927 00:41:31.619760 13161 ts_manager.cc:194] Registered new tserver with Master: 755050658c04498d8cca4927aa8b3103 (127.0.98.3:42475)
I20260927 00:41:31.620678 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53557
I20260927 00:41:31.622593 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:31.635007 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.624343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.98.62:44059)
0927 00:41:31.625035 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.625040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.625379 (+ 339us) server_negotiation.cc:407] Connection header received
0927 00:41:31.625426 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.625429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.625474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.625601 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:31.626438 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.627335 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.628689 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.629066 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.629153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.630976 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:31.630980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:31.630983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:31.631016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:31.632950 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:31.633494 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:31.633498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:31.633499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:31.633617 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:31.633916 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:31.633919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:31.633921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:31.634165 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:31.634291 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.634673 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.634829 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":114,"threads_started":1}
I20260927 00:41:31.639586 13161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Sep 27 00:41:31 dist-test-slave-xltq krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:41:31.645308 13161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:31.670967 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.661399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.0.98.1:45115)
0927 00:41:31.662062 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.662071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.663628 (+ 1557us) server_negotiation.cc:407] Connection header received
0927 00:41:31.663958 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.663965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.664301 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.664437 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:31.665022 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.666196 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.667975 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.669561 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.669815 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.670008 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.670661 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.670798 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":161,"threads_started":1}
I20260927 00:41:31.671787 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.660281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:57352)
0927 00:41:31.663433 (+ 3152us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.663513 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:31.663765 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:31.664698 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:31.664706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:31.664734 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:31.664885 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.664893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.666354 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.666357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.667816 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.667829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.670267 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.670270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.670404 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.670455 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 00:41:31.671455 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0927 00:41:31.671637 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2799,"thread_start_us":144,"threads_started":1}
I20260927 00:41:31.672381 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.662198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42475 (local address 127.0.0.1:51380)
0927 00:41:31.664986 (+ 2788us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.665001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:31.665091 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:31.665848 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:31.665851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:31.665869 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:31.665965 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.665971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.668384 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.668387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.669541 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.669552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.671965 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.671968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.672057 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.672081 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:41:31.672164 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:41:31.672263 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2726,"thread_start_us":120,"threads_started":1}
I20260927 00:41:31.673079 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.662433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.0.98.3:42475)
0927 00:41:31.662753 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.662762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.665201 (+ 2439us) server_negotiation.cc:407] Connection header received
0927 00:41:31.665291 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.665296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.665532 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.665672 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:31.667061 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.668245 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.669693 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.671631 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.671740 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.671897 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.672817 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.672942 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260927 00:41:31.675879 13291 tablet_service.cc:1467] Processing CreateTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 (DEFAULT_TABLE table=test-table [id=354827d69ecb47d786b5ba04abafc9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:31.677505 13291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b760255b1fa4bd6a4de840d226225d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:31.677495 13562 tablet_service.cc:1467] Processing CreateTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 (DEFAULT_TABLE table=test-table [id=354827d69ecb47d786b5ba04abafc9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:31.679371 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b760255b1fa4bd6a4de840d226225d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:31.680109 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.663930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33135 (local address 127.0.0.1:33622)
0927 00:41:31.668456 (+ 4526us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.668471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:31.668982 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:31.674461 (+ 5479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:31.674465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:31.674495 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:31.674692 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.674705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.676186 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.676190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.677560 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.677577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.679581 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.679586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.679718 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.679749 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:41:31.679829 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:41:31.679934 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4441,"thread_start_us":98,"threads_started":1}
I20260927 00:41:31.680778 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.664390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.0.98.2:33135)
0927 00:41:31.670142 (+ 5752us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.670150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.670180 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:41:31.674040 (+ 3860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.674046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.674292 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.674455 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:31.674850 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.676042 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.677748 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.679190 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.679323 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.679491 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.680503 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.680627 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5309,"thread_start_us":124,"threads_started":1}
I20260927 00:41:31.684075 13415 tablet_service.cc:1467] Processing CreateTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 (DEFAULT_TABLE table=test-table [id=354827d69ecb47d786b5ba04abafc9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:31.685324 13415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b760255b1fa4bd6a4de840d226225d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:31.688468 13667 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Bootstrap starting.
I20260927 00:41:31.691112 13667 tablet_bootstrap.cc:654] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:31.692139 13667 log.cc:824] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:31.694733 13667 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: No bootstrap required, opened a new log
I20260927 00:41:31.694973 13667 ts_tablet_manager.cc:1397] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:41:31.697418 13669 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Bootstrap starting.
I20260927 00:41:31.699309 13667 raft_consensus.cc:348] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.699695 13667 raft_consensus.cc:374] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:31.699766 13667 raft_consensus.cc:729] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df111d9c5c54442f97789140a4ae5bc9, State: Initialized, Role: FOLLOWER
I20260927 00:41:31.700098 13669 tablet_bootstrap.cc:654] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:31.700402 13667 consensus_queue.cc:260] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.701167 13669 log.cc:824] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:31.701797 13667 ts_tablet_manager.cc:1428] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:41:31.702306 13670 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Bootstrap starting.
I20260927 00:41:31.704439 13365 heartbeater.cc:503] Master 127.0.98.62:44059 was elected leader, sending a full tablet report...
I20260927 00:41:31.704820 13670 tablet_bootstrap.cc:654] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:31.705606 13670 log.cc:824] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:31.708247 13669 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: No bootstrap required, opened a new log
I20260927 00:41:31.708499 13669 ts_tablet_manager.cc:1397] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 00:41:31.710407 13670 tablet_bootstrap.cc:492] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: No bootstrap required, opened a new log
I20260927 00:41:31.710711 13670 ts_tablet_manager.cc:1397] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 00:41:31.712909 13669 raft_consensus.cc:348] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.713363 13671 raft_consensus.cc:482] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:31.713582 13671 raft_consensus.cc:504] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.715984 13669 raft_consensus.cc:374] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:31.716133 13669 raft_consensus.cc:729] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d596a6216c994323adc15ad0ccebcf12, State: Initialized, Role: FOLLOWER
I20260927 00:41:31.716537 13670 raft_consensus.cc:348] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.716913 13670 raft_consensus.cc:374] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:31.716997 13670 raft_consensus.cc:729] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755050658c04498d8cca4927aa8b3103, State: Initialized, Role: FOLLOWER
I20260927 00:41:31.717474 13670 consensus_queue.cc:260] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.716781 13669 consensus_queue.cc:260] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.718106 13671 leader_election.cc:288] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 755050658c04498d8cca4927aa8b3103 (127.0.98.3:42475), d596a6216c994323adc15ad0ccebcf12 (127.0.98.2:33135)
I20260927 00:41:31.718720 13670 ts_tablet_manager.cc:1428] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:41:31.719141 13633 heartbeater.cc:503] Master 127.0.98.62:44059 was elected leader, sending a full tablet report...
I20260927 00:41:31.720242 13669 ts_tablet_manager.cc:1428] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 00:41:31.720646 13499 heartbeater.cc:503] Master 127.0.98.62:44059 was elected leader, sending a full tablet report...
I20260927 00:41:31.727823 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.715775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45049 (local address 127.0.98.3:42475)
0927 00:41:31.715895 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.715899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.722185 (+ 6286us) server_negotiation.cc:407] Connection header received
0927 00:41:31.722603 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.722607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.722653 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.722711 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:31.723661 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.724395 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.725614 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.726177 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.726265 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.726331 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.727595 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.727683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:31.728268 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.715339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42475 (local address 127.0.98.1:45049)
0927 00:41:31.722095 (+ 6756us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.722113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:31.722306 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:31.723289 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:31.723292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:31.723364 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:31.723516 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.723525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.724620 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.724623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.725493 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.725504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.726384 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.726387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.727450 (+ 1063us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.727477 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:41:31.727988 (+ 511us) client_negotiation.cc:240] Negotiation successful
0927 00:41:31.728075 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6663,"thread_start_us":128,"threads_started":1}
I20260927 00:41:31.729041 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b760255b1fa4bd6a4de840d226225d6" candidate_uuid: "df111d9c5c54442f97789140a4ae5bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755050658c04498d8cca4927aa8b3103" is_pre_election: true
I20260927 00:41:31.729401 13586 raft_consensus.cc:2454] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df111d9c5c54442f97789140a4ae5bc9 in term 0.
I20260927 00:41:31.729977 13254 leader_election.cc:302] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755050658c04498d8cca4927aa8b3103, df111d9c5c54442f97789140a4ae5bc9; no voters:
I20260927 00:41:31.730306 13671 raft_consensus.cc:2781] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:31.730479 13671 raft_consensus.cc:482] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:31.730535 13671 raft_consensus.cc:3037] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:31.730700 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.717763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46991 (local address 127.0.98.2:33135)
0927 00:41:31.717898 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.717902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.725877 (+ 7975us) server_negotiation.cc:407] Connection header received
0927 00:41:31.726045 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.726048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.726094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.726139 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:31.726515 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.727151 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.729047 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.729623 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.729736 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.729803 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.730466 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.730564 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:41:31.731119 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.717352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33135 (local address 127.0.98.1:46991)
0927 00:41:31.725797 (+ 8445us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.725811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:31.725944 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:31.726270 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:31.726273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:31.726291 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:31.726398 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.726404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.727278 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.727280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.728898 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:31.728910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.730197 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.730201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:31.730306 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.730334 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:41:31.730889 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 00:41:31.730974 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8360,"thread_start_us":122,"threads_started":1}
I20260927 00:41:31.731786 13453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b760255b1fa4bd6a4de840d226225d6" candidate_uuid: "df111d9c5c54442f97789140a4ae5bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d596a6216c994323adc15ad0ccebcf12" is_pre_election: true
I20260927 00:41:31.731894 13671 raft_consensus.cc:504] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.732100 13453 raft_consensus.cc:2454] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df111d9c5c54442f97789140a4ae5bc9 in term 0.
I20260927 00:41:31.732695 13671 leader_election.cc:288] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 755050658c04498d8cca4927aa8b3103 (127.0.98.3:42475), d596a6216c994323adc15ad0ccebcf12 (127.0.98.2:33135)
I20260927 00:41:31.733084 13453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b760255b1fa4bd6a4de840d226225d6" candidate_uuid: "df111d9c5c54442f97789140a4ae5bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d596a6216c994323adc15ad0ccebcf12"
I20260927 00:41:31.733091 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b760255b1fa4bd6a4de840d226225d6" candidate_uuid: "df111d9c5c54442f97789140a4ae5bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755050658c04498d8cca4927aa8b3103"
I20260927 00:41:31.733209 13453 raft_consensus.cc:3037] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:31.733234 13586 raft_consensus.cc:3037] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:31.734393 13453 raft_consensus.cc:2454] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df111d9c5c54442f97789140a4ae5bc9 in term 1.
I20260927 00:41:31.734596 13586 raft_consensus.cc:2454] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df111d9c5c54442f97789140a4ae5bc9 in term 1.
I20260927 00:41:31.734815 13253 leader_election.cc:302] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d596a6216c994323adc15ad0ccebcf12, df111d9c5c54442f97789140a4ae5bc9; no voters:
I20260927 00:41:31.735116 13671 raft_consensus.cc:2781] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:31.735365 13671 raft_consensus.cc:686] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [term 1 LEADER]: Becoming Leader. State: Replica: df111d9c5c54442f97789140a4ae5bc9, State: Running, Role: LEADER
I20260927 00:41:31.735780 13671 consensus_queue.cc:237] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } }
I20260927 00:41:31.740123 13161 catalog_manager.cc:5495] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to df111d9c5c54442f97789140a4ae5bc9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "df111d9c5c54442f97789140a4ae5bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df111d9c5c54442f97789140a4ae5bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:31.791275 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.785276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.98.1:45115)
0927 00:41:31.786228 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:31.786232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:31.786247 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:31.786351 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:31.786354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:31.786428 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:31.786527 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:31.786896 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.787747 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.789032 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:31.789414 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:31.789507 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:31.789943 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:31.790582 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:31.790695 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:31.791032 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:41:31.791096 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848}
I20260927 00:41:31.808835 13453 raft_consensus.cc:1261] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 1 FOLLOWER]: Refusing update from remote peer df111d9c5c54442f97789140a4ae5bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:31.808835 13586 raft_consensus.cc:1261] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 1 FOLLOWER]: Refusing update from remote peer df111d9c5c54442f97789140a4ae5bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:31.809684 13679 consensus_queue.cc:1035] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755050658c04498d8cca4927aa8b3103" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:31.810547 13679 consensus_queue.cc:1035] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d596a6216c994323adc15ad0ccebcf12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:41:31.813846 13366 tablet.cc:2369] T 8b760255b1fa4bd6a4de840d226225d6 P df111d9c5c54442f97789140a4ae5bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:31.822769 13682 mvcc.cc:205] Tried to move back new op lower bound from 7333763857624043520 to 7333763857355526144. Current Snapshot: MvccSnapshot[applied={T|T < 7333763857624043520}]
I20260927 00:41:31.852357 13162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:31.852568 13162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:31.855744 13162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d38c262c6964eb4a8bd5c88a236a574: Sending DeleteTablet for 3 replicas of tablet 8b760255b1fa4bd6a4de840d226225d6
I20260927 00:41:31.856875 13562 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51380
I20260927 00:41:31.857102 13415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33622
I20260927 00:41:31.857453 13291 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b760255b1fa4bd6a4de840d226225d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57352
I20260927 00:41:31.858781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13205
I20260927 00:41:31.858755 13685 mvcc.cc:205] Tried to move back new op lower bound from 7333763857624043520 to 7333763857355526144. Current Snapshot: MvccSnapshot[applied={T|T < 7333763857624043520}]
I20260927 00:41:31.862938 13692 tablet_replica.cc:330] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: stopping tablet replica
I20260927 00:41:31.863409 13692 raft_consensus.cc:2229] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:31.863843 13692 raft_consensus.cc:2258] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:31.870445 13692 ts_tablet_manager.cc:1905] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:31.870878 13691 tablet_replica.cc:330] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: stopping tablet replica
I20260927 00:41:31.871279 13691 raft_consensus.cc:2229] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:31.874265 13692 ts_tablet_manager.cc:1918] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:31.874809 13691 raft_consensus.cc:2258] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:31.875150 13692 log.cc:1196] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/8b760255b1fa4bd6a4de840d226225d6
I20260927 00:41:31.876724 13691 ts_tablet_manager.cc:1905] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:31.879750 13691 ts_tablet_manager.cc:1918] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:31.880359 13692 ts_tablet_manager.cc:1939] T 8b760255b1fa4bd6a4de840d226225d6 P d596a6216c994323adc15ad0ccebcf12: Deleting consensus metadata
W20260927 00:41:31.881129 13154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45115 (error 108)
I20260927 00:41:31.883510 13152 catalog_manager.cc:4841] TS d596a6216c994323adc15ad0ccebcf12 (127.0.98.2:33135): tablet 8b760255b1fa4bd6a4de840d226225d6 (table test-table [id=354827d69ecb47d786b5ba04abafc9d8]) successfully deleted
I20260927 00:41:31.883965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13369
I20260927 00:41:31.885329 13691 log.cc:1196] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/8b760255b1fa4bd6a4de840d226225d6
I20260927 00:41:31.885828 13691 ts_tablet_manager.cc:1939] T 8b760255b1fa4bd6a4de840d226225d6 P 755050658c04498d8cca4927aa8b3103: Deleting consensus metadata
I20260927 00:41:31.887480 13153 catalog_manager.cc:4841] TS 755050658c04498d8cca4927aa8b3103 (127.0.98.3:42475): tablet 8b760255b1fa4bd6a4de840d226225d6 (table test-table [id=354827d69ecb47d786b5ba04abafc9d8]) successfully deleted
I20260927 00:41:31.891481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13501
I20260927 00:41:31.892549 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:31.892177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45115 (local address 127.0.0.1:57360)
0927 00:41:31.892355 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:31.892448 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:41:31.892796 13154 catalog_manager.cc:4584] TS df111d9c5c54442f97789140a4ae5bc9 (127.0.98.1:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b760255b1fa4bd6a4de840d226225d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45115: connect: Connection refused (error 111)
I20260927 00:41:31.899966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13134
I20260927 00:41:31.909047 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:41:32.244694 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:41:32.244860 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:41:32.581801 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:41:32.582 UTC [12201] LOG: received smart shutdown request
2026-09-27 00:41:32.585 UTC [12201] LOG: background worker "logical replication launcher" (PID 12210) exited with exit code 1
2026-09-27 00:41:32.586 UTC [12205] LOG: shutting down
2026-09-27 00:41:32.794 UTC [12201] LOG: database system is shut down
2026-09-27T00:41:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51223 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:32 dist-test-slave-xltq krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:32 dist-test-slave-xltq krb5kdc[13703](info): set up 2 sockets
Sep 27 00:41:32 dist-test-slave-xltq krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260927 00:41:33.868125 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:33 dist-test-slave-xltq krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:33.992916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:34.109865 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:34.110195 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:34.110256 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:34.114669 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:34.114764 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:34.114791 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:34.114813 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:34.114833 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:34.120625 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:34.121840 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:34.122926 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:34.132072 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.132262 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.132076 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:34.132427 13716 server_base.cc:1034] running on GCE node
I20260927 00:41:34.133015 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:34.134220 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:34.135442 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790469694135421 us; error 41 us; skew 500 ppm
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:34.138837 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:34.139918 13716 webserver.cc:460] Webserver started at http://127.0.98.62:33703/ using document root <none> and password file <none>
I20260927 00:41:34.140563 13716 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:34.140645 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:34.140872 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:34.143201 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "595a78d819c4462e98948f86b67f0a5c"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "d2c43055e51d03741df6294d62b65976"
server_key_iv: "650275b26c02b158029c4b16601a3fd0"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.143832 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "595a78d819c4462e98948f86b67f0a5c"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "d2c43055e51d03741df6294d62b65976"
server_key_iv: "650275b26c02b158029c4b16601a3fd0"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.147933 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:41:34.150841 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.152427 13716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:34.152573 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "595a78d819c4462e98948f86b67f0a5c"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "d2c43055e51d03741df6294d62b65976"
server_key_iv: "650275b26c02b158029c4b16601a3fd0"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.152694 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:34.180104 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:34.183835 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:34.184013 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:34.190827 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45149
I20260927 00:41:34.190853 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45149 every 8 connection(s)
I20260927 00:41:34.191807 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:34.194769 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:34.198855 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.200132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13716
I20260927 00:41:34.200277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:34.200601 392 external_mini_cluster.cc:1428] Setting key f8ee1a7fcf37295e37dc0367489c735c
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:34.219228 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.201463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.0.98.62:45149)
0927 00:41:34.202080 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.202099 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.202316 (+ 217us) server_negotiation.cc:407] Connection header received
0927 00:41:34.203745 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.203750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.203950 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.204117 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.205296 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.206325 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.207899 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.208347 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.208488 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.213390 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.213399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.213418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.213455 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.215187 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.216112 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.216121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.216123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.216308 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.216825 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.216828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.216830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.217349 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.217472 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.217991 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.218318 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":481,"spinlock_wait_cycles":30592,"thread_start_us":164,"threads_started":1}
I20260927 00:41:34.219228 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.201463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45149 (local address 127.0.0.1:36142)
0927 00:41:34.202118 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.202182 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.203365 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.204162 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.204168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.204708 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.205093 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.205113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.206464 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.206468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.207728 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.207742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.208537 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.208542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.208684 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.209286 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.209304 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.213138 (+ 3834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.215358 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.215371 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.215385 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.215886 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.216499 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.216503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.216506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.216694 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.217485 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.217494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.217711 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.217941 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.218400 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":117,"thread_start_us":147,"threads_started":1}
I20260927 00:41:34.222261 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.223189 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.222837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53724)
0927 00:41:34.222979 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.223062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:41:34.223393 13733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
I20260927 00:41:34.224546 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.224291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53736)
0927 00:41:34.224417 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.224471 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:41:34.224704 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:41:34.232192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33385
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:34.273999 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } attempt: 1
I20260927 00:41:34.274919 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.274610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53746)
0927 00:41:34.274771 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.274834 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 00:41:34.275604 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.275387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53762)
0927 00:41:34.275490 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.275536 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:41:34.275729 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
I20260927 00:41:34.343892 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } attempt: 2
I20260927 00:41:34.344825 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.344501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53772)
0927 00:41:34.344669 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.344739 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:41:34.345654 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.345353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53782)
0927 00:41:34.345538 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.345589 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:41:34.345773 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33385: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33385: connect: Connection refused (error 111)
W20260927 00:41:34.348146 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:34.348501 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:34.348572 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:34.353159 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:34.353255 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:34.353283 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:34.353307 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:34.353335 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:34.359141 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:34.360383 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:34.361691 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:34.370699 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.370711 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.370712 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:34.371011 13791 server_base.cc:1034] running on GCE node
I20260927 00:41:34.371754 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:34.372992 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:34.374193 13791 hybrid_clock.cc:648] HybridClock initialized: now 1790469694374174 us; error 36 us; skew 500 ppm
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:34.377679 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:41:34.378786 13791 webserver.cc:460] Webserver started at http://127.0.98.61:36627/ using document root <none> and password file <none>
I20260927 00:41:34.379398 13791 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:34.379482 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:34.379705 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:34.382119 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data/instance:
uuid: "fc59aeb004364bfaab621bb52aae1864"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "8539f86026a28feb797013e30bcc4ffb"
server_key_iv: "52976d68f6d33d26de432f548e117387"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.383015 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal/instance:
uuid: "fc59aeb004364bfaab621bb52aae1864"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "8539f86026a28feb797013e30bcc4ffb"
server_key_iv: "52976d68f6d33d26de432f548e117387"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.387593 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:41:34.390825 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.392704 13791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:34.392866 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "fc59aeb004364bfaab621bb52aae1864"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "8539f86026a28feb797013e30bcc4ffb"
server_key_iv: "52976d68f6d33d26de432f548e117387"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.392990 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:34.417402 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:34.421578 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:34.421805 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:34.429380 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33385
I20260927 00:41:34.429554 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33385 every 8 connection(s)
I20260927 00:41:34.430462 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:41:34.433542 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:34.436944 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } attempt: 3
I20260927 00:41:34.438089 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.439419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13791
I20260927 00:41:34.439548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-1/wal/instance
I20260927 00:41:34.439814 392 external_mini_cluster.cc:1428] Setting key af13d24a0c88a5c1535a39c921e665d1
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 00:41:34.450521 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.437498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53794)
0927 00:41:34.437645 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.437663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.437760 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.440324 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.440327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.440516 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.440670 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.440678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.442448 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.442452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.443347 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.443359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.444437 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.444440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.444545 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.445034 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.445048 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.446473 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.448479 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.448482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.448484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.448716 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.449125 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.449129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.449130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.449220 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.449999 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.450002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.450117 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.450228 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.450375 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 00:41:34.451768 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.440145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.0.98.62:45149)
0927 00:41:34.441457 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.441462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.441476 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:34.441514 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.441517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.441561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.441643 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.442454 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.443137 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.444399 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.444783 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.444893 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.447986 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.447989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.447992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.448019 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.449462 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.449947 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.449951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.449953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.450040 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.450312 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.450315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.450317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.450512 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.450602 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.451549 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.451652 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":1206}
I20260927 00:41:34.451848 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.438255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.0.98.61:33385)
0927 00:41:34.438784 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.438800 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.438846 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:41:34.439512 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.439535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.439971 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.440402 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.440821 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.442236 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.443466 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.444270 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.444435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.446633 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.446641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.446656 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.446683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.448371 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.448832 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.448840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.448847 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.448991 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.449304 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.449308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.449309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.449820 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.449978 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.450424 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.450774 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":387,"thread_start_us":141,"threads_started":1}
I20260927 00:41:34.451848 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.440144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45149 (local address 127.0.0.1:36150)
0927 00:41:34.441108 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.441170 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.441273 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.441704 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.441709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.442167 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.442321 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.442339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.443276 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.443279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.444259 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.444269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.445533 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.445542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.445631 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.446112 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.446133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.447833 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.449595 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.449603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.449607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.449845 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.450127 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.450130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.450132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.450221 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.450651 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.450657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.450767 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.450921 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.451088 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":1,"thread_start_us":181,"threads_started":1}
I20260927 00:41:34.453920 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.453930 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.454717 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.454406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50118)
0927 00:41:34.454593 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.454652 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 00:41:34.455544 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.455285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50126)
0927 00:41:34.455426 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.455478 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 00:41:34.455684 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:41:34.464409 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.454697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53798)
0927 00:41:34.455221 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.455234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.455318 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.455547 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.455549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.455823 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.455970 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.455978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.457048 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.457052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.457704 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.457713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.458290 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.458293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.458446 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.458992 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.459004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.460373 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.462548 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.462560 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.462563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.462798 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.463085 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.463088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.463090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.463176 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.463632 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.463636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.463759 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.463980 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.464176 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459}
I20260927 00:41:34.464530 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.454863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.98.61:33385)
0927 00:41:34.454966 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.454971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.455397 (+ 426us) server_negotiation.cc:407] Connection header received
0927 00:41:34.455422 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.455425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.455458 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.455517 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.456089 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.456921 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.457822 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.458161 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.458267 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.460538 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.460541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.460543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.460575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.462416 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.462885 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.462889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.462891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.462970 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.463268 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.463271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.463273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.463502 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.463600 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.464118 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.464354 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:41:34.465466 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.466228 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.465945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50142)
0927 00:41:34.466060 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.466151 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:41:34.466801 13807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
I20260927 00:41:34.467796 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.467569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50146)
0927 00:41:34.467687 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.467738 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:41:34.467942 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:41:34.469558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34135
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:34.486846 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 1
I20260927 00:41:34.488148 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.487585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50154)
0927 00:41:34.487906 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.488035 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 00:41:34.488862 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.488657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50156)
0927 00:41:34.488754 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.488802 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:41:34.489006 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
I20260927 00:41:34.517238 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 1
I20260927 00:41:34.518208 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.517871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50168)
0927 00:41:34.518055 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.518123 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 00:41:34.519143 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.518886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50174)
0927 00:41:34.519021 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.519077 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:41:34.519261 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
I20260927 00:41:34.564239 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 2
I20260927 00:41:34.565191 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.564837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50184)
0927 00:41:34.565009 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.565089 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:41:34.565954 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.565714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50186)
0927 00:41:34.565825 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.565882 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:41:34.566118 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
I20260927 00:41:34.587435 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 2
I20260927 00:41:34.588315 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.587982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50188)
0927 00:41:34.588142 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.588230 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 00:41:34.588311 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:34.588671 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:34.588732 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:41:34.589197 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.588967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50190)
0927 00:41:34.589077 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.589128 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:41:34.589381 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34135: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34135: connect: Connection refused (error 111)
W20260927 00:41:34.593201 13866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:34.593273 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:34.593302 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:34.593333 13866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:34.593361 13866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:34.598995 13866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45149,127.0.98.61:33385,127.0.98.60:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:34.600116 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:34.601154 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:34.610289 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.610309 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.610280 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:34.611027 13866 server_base.cc:1034] running on GCE node
I20260927 00:41:34.611764 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:34.613063 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:34.614239 13866 hybrid_clock.cc:648] HybridClock initialized: now 1790469694614218 us; error 43 us; skew 500 ppm
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:34.617319 13866 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:41:34.618340 13866 webserver.cc:460] Webserver started at http://127.0.98.60:43523/ using document root <none> and password file <none>
I20260927 00:41:34.618928 13866 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:34.619009 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:34.619225 13866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:34.621510 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data/instance:
uuid: "79afee625a9d41748b023f08f08da4e6"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "4235373e128bfd0f5c12321d5e706176"
server_key_iv: "ce97d92cb7319eb07ba7593721cc620f"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.622144 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal/instance:
uuid: "79afee625a9d41748b023f08f08da4e6"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "4235373e128bfd0f5c12321d5e706176"
server_key_iv: "ce97d92cb7319eb07ba7593721cc620f"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.626339 13866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:41:34.629014 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.630561 13866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:34.630717 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "79afee625a9d41748b023f08f08da4e6"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-xltq"
server_key: "4235373e128bfd0f5c12321d5e706176"
server_key_iv: "ce97d92cb7319eb07ba7593721cc620f"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.630846 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:34.645028 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:34.648746 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:34.648979 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:34.656015 13866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34135
I20260927 00:41:34.656054 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34135 every 8 connection(s)
I20260927 00:41:34.657117 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:41:34.660163 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:34.664520 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.665287 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 3
I20260927 00:41:34.666626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 13866
I20260927 00:41:34.666750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469528995939-392-0/minicluster-data/master-2/wal/instance
I20260927 00:41:34.667038 392 external_mini_cluster.cc:1428] Setting key 681f1d1438a1d72576381837745a4b5c
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:41:34.678949 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.666439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.0.98.62:45149)
0927 00:41:34.666542 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.666547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.667386 (+ 839us) server_negotiation.cc:407] Connection header received
0927 00:41:34.668421 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.668424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.668469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.668567 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.669557 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.670400 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.671773 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.671998 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.672084 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.674740 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.674751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.674754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.674887 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.676436 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677058 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.677063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.677065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.677158 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677590 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.677593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.677595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.677996 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.678109 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.678609 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.678752 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":768}
I20260927 00:41:34.679586 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.665800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50192)
0927 00:41:34.665969 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.665988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.666097 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.668633 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.668636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.668845 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.668970 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.668977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.670379 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.670383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.671181 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.671193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.672142 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.672146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.672255 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.672905 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.672918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.674890 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.677104 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.677110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.677359 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.677649 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.677654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.677737 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.678347 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.678350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.678511 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.678606 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.679434 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":661}
I20260927 00:41:34.680173 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.666635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45149 (local address 127.0.0.1:36166)
0927 00:41:34.667247 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.667297 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.668139 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.668626 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.668632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.669066 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.669418 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.669431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.670510 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.670513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.671665 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.671674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.672145 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.672149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.672275 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.672892 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.672923 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.674515 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.676562 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.676571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.676582 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.676946 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.677241 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.677246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.677465 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.678171 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.678179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.678351 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.678550 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.678983 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":85,"spinlock_wait_cycles":47360,"thread_start_us":189,"threads_started":1}
I20260927 00:41:34.680173 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.666634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.60:34135)
0927 00:41:34.667101 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.667120 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.667161 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:41:34.668119 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.668147 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.668469 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.669086 (+ 617us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.669248 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.670263 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.671306 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.672004 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.672525 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.675063 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.675071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.675086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.675114 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.677021 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677474 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.677480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.677482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.677559 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.677819 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.677822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.677824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.678248 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.678381 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.678791 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.679047 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":217,"server-negotiator.queue_time_us":334,"spinlock_wait_cycles":28544,"thread_start_us":151,"threads_started":1}
I20260927 00:41:34.680550 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.667744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.98.60:34135)
0927 00:41:34.668017 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.668024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.668037 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:34.668176 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.668179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.668469 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.668626 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.669566 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.670426 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.671231 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.672004 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.672158 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.675586 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.675590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.675592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.675619 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.678160 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.678671 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.678674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.678676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.678755 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.679060 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.679064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.679066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.679261 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.679449 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.680127 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.680356 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":7808,"thread_start_us":101,"threads_started":1}
I20260927 00:41:34.680713 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } attempt: 3
I20260927 00:41:34.681818 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.684849 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: Bootstrap starting.
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:41:34.689671 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: Neither blocks nor log segments found. Creating new log.
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:34.690687 13784 log.cc:824] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:34.691417 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.681172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50200)
0927 00:41:34.681306 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.681331 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.681425 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.682562 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.682565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.682756 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.682897 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.682905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.683885 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.683889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.684524 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.684536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.685096 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.685101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.685193 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.685718 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.685733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.687261 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.689302 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.689305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.689307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.689905 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.690215 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.690220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.690310 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.690780 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.690783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.690933 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.691090 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.691241 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:41:34.691843 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.682597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.98.61:33385)
0927 00:41:34.682703 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.682708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.682841 (+ 133us) server_negotiation.cc:407] Connection header received
0927 00:41:34.683014 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.683017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.683061 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.683107 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.683648 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.684369 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.685372 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.685719 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.685789 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.688179 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.688183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.688185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.688249 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.689699 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690464 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.690468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.690469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.690553 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690875 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.690878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.690888 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.691066 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.691124 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.691461 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.691707 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:41:34.691994 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.682103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.98.60:34135)
0927 00:41:34.682329 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.682333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.682345 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:34.682379 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.682382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.682425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.683021 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.683107 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.683784 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.684652 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.684961 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.685437 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.687404 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.687407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.687410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.687435 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.689182 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690000 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.690004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.690006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.690105 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690420 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.690422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.690424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.690648 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.691612 (+ 964us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.691687 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.691807 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 00:41:34.692283 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.682368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33385 (local address 127.0.0.1:53806)
0927 00:41:34.682781 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.682796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.682898 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.683220 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.683222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.683404 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.683510 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.683516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.684500 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.684503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.685236 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.685247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.686024 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.686028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.686121 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.686536 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.686550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.688008 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.690015 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.690020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.690348 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.690657 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.690659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.690661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.690746 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.691194 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.691196 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.691287 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.691972 (+ 685us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.692111 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260927 00:41:34.694480 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: No bootstrap required, opened a new log
I20260927 00:41:34.694512 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.684390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.0.98.62:45149)
0927 00:41:34.684500 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.684505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.684517 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:34.684553 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.684555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.684600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.684664 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.685508 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.686202 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.687296 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.687622 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.687696 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.690646 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.690650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.690653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.690679 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.692780 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.693261 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.693265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.693267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.693359 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.693639 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.693642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.693643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.693867 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.694032 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.694252 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.694372 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:34.695252 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: Bootstrap starting.
I20260927 00:41:34.697729 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:34.697760 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.697983 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:34.698036 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595a78d819c4462e98948f86b67f0a5c, State: Initialized, Role: FOLLOWER
I20260927 00:41:34.698468 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.698550 13859 log.cc:824] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:34.698988 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.699221 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.699894 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:34.700325 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:34.700961 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: No bootstrap required, opened a new log
W20260927 00:41:34.705799 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:41:34.705653 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
W20260927 00:41:34.705962 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:34.705929 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:34.706168 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc59aeb004364bfaab621bb52aae1864, State: Initialized, Role: FOLLOWER
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:41:34.706805 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.708559 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:34.705905 13716 master_runner.cc:186] Error getting master registration for 127.0.98.61:33385: OK, instance_id { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" instance_seqno: 1790469694427235 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:41:34.710093 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:41:34.711697 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.700790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.98.61:33385)
0927 00:41:34.700914 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.700918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.700932 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:34.700972 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.700975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.701019 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.701891 (+ 872us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.702054 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.702796 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.703981 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.704339 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.704438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.707514 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.707518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.707521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.707552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.709439 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.710012 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.710016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.710018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.710110 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.710413 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.710416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.710419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.710652 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.710826 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.711410 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.711537 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:41:34 dist-test-slave-xltq krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:41:34.717070 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.717226 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.717449 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:41:34.719022 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:34.719110 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:34.719545 13791 master_runner.cc:186] Error getting master registration for 127.0.98.60:34135: OK, instance_id { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" instance_seqno: 1790469694653975 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:41:34.723529 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:41:34.728291 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.700839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34135 (local address 127.0.0.1:50204)
0927 00:41:34.705024 (+ 4185us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.705041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.707856 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.708693 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.708696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.710994 (+ 2298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.711123 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.711131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.712400 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.712403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.713525 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.713539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.714393 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.714396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.714477 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.715941 (+ 1464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.715955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.724867 (+ 8912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.726838 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.726841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.726843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.727089 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.727313 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.727316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.727318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.727396 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.727780 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.727783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.727858 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.727957 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.728090 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4080}
I20260927 00:41:34.728849 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.701038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.98.60:34135)
0927 00:41:34.701161 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.701166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.705119 (+ 3953us) server_negotiation.cc:407] Connection header received
0927 00:41:34.708084 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.708087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.708139 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.708185 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.711365 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.712038 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.713651 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.713948 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.714025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.725041 (+ 11016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.725046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.725049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.725078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.726727 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.727174 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.727177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.727178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.727239 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.727474 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.727477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.727478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.727671 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.727749 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.728603 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.728710 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:41:34.732700 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: Bootstrap starting.
I20260927 00:41:34.733309 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.723775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.0.98.62:45149)
0927 00:41:34.723920 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.723925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.723944 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:41:34.724001 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.724004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.724054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.724151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.725513 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.726293 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.727446 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.727758 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.727831 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.729862 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.729866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.729869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.729901 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.731471 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.731930 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.731934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.731935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.732025 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.732340 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.732344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.732346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.732577 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.732686 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.733022 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.733154 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:41:34.734922 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.724350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.98.60:34135)
0927 00:41:34.724468 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.724472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.726395 (+ 1923us) server_negotiation.cc:407] Connection header received
0927 00:41:34.726502 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.726505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.726555 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.726628 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.727514 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.728296 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.729330 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.729693 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.729764 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.731449 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.731453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.731456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.731481 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.732997 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.733435 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.733439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.733441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.733527 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.733843 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.733847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.733849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.734072 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.734180 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.734549 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.734754 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:41:34.736415 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:34.736615 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.725292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.98.61:33385)
0927 00:41:34.725425 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.725430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.726668 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 00:41:34.726832 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.726836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.726883 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.726933 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.727846 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.728891 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.729919 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.730878 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.730974 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.733004 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.733007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.733010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.733039 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.734601 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.735177 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.735181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.735183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.735277 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.735578 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.735582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.735584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.735792 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.735888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.736257 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.736470 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:41:34.737660 13934 log.cc:824] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:34.740118 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: No bootstrap required, opened a new log
I20260927 00:41:34.744428 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.744786 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:34.744860 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79afee625a9d41748b023f08f08da4e6, State: Initialized, Role: FOLLOWER
I20260927 00:41:34.745436 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.747046 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:34.747570 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.747759 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:41:34.755309 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:34.755544 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:34.756373 13866 master_runner.cc:263] No leader master found from master 79afee625a9d41748b023f08f08da4e6
I20260927 00:41:34.756445 13866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:41:34.863935 13976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:34.864162 13976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.865330 13976 leader_election.cc:288] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595a78d819c4462e98948f86b67f0a5c (127.0.98.62:45149), fc59aeb004364bfaab621bb52aae1864 (127.0.98.61:33385)
I20260927 00:41:34.866307 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afee625a9d41748b023f08f08da4e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595a78d819c4462e98948f86b67f0a5c" is_pre_election: true
I20260927 00:41:34.866638 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79afee625a9d41748b023f08f08da4e6 in term 0.
I20260927 00:41:34.867264 13884 leader_election.cc:302] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595a78d819c4462e98948f86b67f0a5c, 79afee625a9d41748b023f08f08da4e6; no voters:
I20260927 00:41:34.867659 13976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:34.867741 13976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:34.867825 13976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:34.868983 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afee625a9d41748b023f08f08da4e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc59aeb004364bfaab621bb52aae1864" is_pre_election: true
I20260927 00:41:34.869302 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79afee625a9d41748b023f08f08da4e6 in term 0.
I20260927 00:41:34.870726 13976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.871457 13976 leader_election.cc:288] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [CANDIDATE]: Term 1 election: Requested vote from peers 595a78d819c4462e98948f86b67f0a5c (127.0.98.62:45149), fc59aeb004364bfaab621bb52aae1864 (127.0.98.61:33385)
I20260927 00:41:34.871997 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afee625a9d41748b023f08f08da4e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595a78d819c4462e98948f86b67f0a5c"
I20260927 00:41:34.872136 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:34.873584 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79afee625a9d41748b023f08f08da4e6 in term 1.
I20260927 00:41:34.874912 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79afee625a9d41748b023f08f08da4e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc59aeb004364bfaab621bb52aae1864"
I20260927 00:41:34.875062 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:34.876564 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79afee625a9d41748b023f08f08da4e6 in term 1.
I20260927 00:41:34.874068 13884 leader_election.cc:302] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595a78d819c4462e98948f86b67f0a5c, 79afee625a9d41748b023f08f08da4e6; no voters:
I20260927 00:41:34.877435 13976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:34.877731 13976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [term 1 LEADER]: Becoming Leader. State: Replica: 79afee625a9d41748b023f08f08da4e6, State: Running, Role: LEADER
I20260927 00:41:34.878182 13976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } }
I20260927 00:41:34.885344 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afee625a9d41748b023f08f08da4e6. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.885609 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:34.886137 13991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:34.887252 13991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:34.894642 13832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [term 1 FOLLOWER]: Refusing update from remote peer 79afee625a9d41748b023f08f08da4e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:34.895501 13989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:41:34.901141 13887 tablet.cc:2369] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:34.904999 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afee625a9d41748b023f08f08da4e6. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.905161 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.907315 13758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [term 1 FOLLOWER]: Refusing update from remote peer 79afee625a9d41748b023f08f08da4e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:34.908121 13976 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:34.913364 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79afee625a9d41748b023f08f08da4e6. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.913543 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.920969 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.921133 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:34.922812 13992 mvcc.cc:205] Tried to move back new op lower bound from 7333763870280884224 to 7333763870239490048. Current Snapshot: MvccSnapshot[applied={T|T < 7333763870280884224}]
W20260927 00:41:34.923624 13812 tablet.cc:2369] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:34.925009 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.925132 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.925972 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.926092 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:34.926121 13995 mvcc.cc:205] Tried to move back new op lower bound from 7333763870280884224 to 7333763870239490048. Current Snapshot: MvccSnapshot[applied={T|T < 7333763870280884224}]
I20260927 00:41:34.926376 13991 catalog_manager.cc:1347] Generated new cluster ID: f089ef1946f44d45aafd1258c54a9a3f
I20260927 00:41:34.926424 13991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:34.928378 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79afee625a9d41748b023f08f08da4e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595a78d819c4462e98948f86b67f0a5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45149 } } peers { permanent_uuid: "fc59aeb004364bfaab621bb52aae1864" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33385 } } peers { permanent_uuid: "79afee625a9d41748b023f08f08da4e6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34135 } } }
I20260927 00:41:34.928506 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:34.928691 13993 mvcc.cc:205] Tried to move back new op lower bound from 7333763870280884224 to 7333763870239490048. Current Snapshot: MvccSnapshot[applied={T|T < 7333763870280884224}]
I20260927 00:41:34.940728 13991 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:34.942166 13991 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:34.952392 13991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79afee625a9d41748b023f08f08da4e6: Generated new TSK 0
I20260927 00:41:34.953478 13991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:35.707265 13953 catalog_manager.cc:1259] Loaded cluster ID: f089ef1946f44d45aafd1258c54a9a3f
I20260927 00:41:35.707355 13953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: loading cluster ID for follower catalog manager: success
I20260927 00:41:35.709316 13953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: acquiring CA information for follower catalog manager: success
I20260927 00:41:35.710744 13953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 595a78d819c4462e98948f86b67f0a5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:41:35.720494 13966 catalog_manager.cc:1259] Loaded cluster ID: f089ef1946f44d45aafd1258c54a9a3f
I20260927 00:41:35.720582 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: loading cluster ID for follower catalog manager: success
I20260927 00:41:35.722380 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: acquiring CA information for follower catalog manager: success
I20260927 00:41:35.723730 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc59aeb004364bfaab621bb52aae1864: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:41:36.022362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13716
I20260927 00:41:36.028352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13791
I20260927 00:41:36.033499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 13866
2026-09-27T00:41:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3147 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:36 dist-test-slave-xltq krb5kdc[14006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:36 dist-test-slave-xltq krb5kdc[14006](info): set up 2 sockets
Sep 27 00:41:36 dist-test-slave-xltq krb5kdc[14006](info): commencing operation
krb5kdc: starting...
W20260927 00:41:37.004285 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:37.131704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37833
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:37.245826 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:37.246130 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:37.246191 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:37.250459 14019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:37.250540 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:37.250568 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:37.250589 14019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:37.250610 14019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:37.256418 14019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:37.257584 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:37.258626 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:37.267782 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.267831 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.267867 14019 server_base.cc:1034] running on GCE node
W20260927 00:41:37.268190 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.268850 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:37.270139 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:37.271323 14019 hybrid_clock.cc:648] HybridClock initialized: now 1790469697271260 us; error 79 us; skew 500 ppm
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:37.274196 14019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:37.275206 14019 webserver.cc:460] Webserver started at http://127.0.98.62:46095/ using document root <none> and password file <none>
I20260927 00:41:37.275769 14019 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:37.275836 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:37.276029 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:37.278278 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "d68237e0b86e4753a21d66fa2c66e362"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-xltq"
server_key: "fe3f8663c6c97d45ce9250de15c8b550"
server_key_iv: "f7187d86263770ad83899243b23b5bf9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.278872 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "d68237e0b86e4753a21d66fa2c66e362"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-xltq"
server_key: "fe3f8663c6c97d45ce9250de15c8b550"
server_key_iv: "f7187d86263770ad83899243b23b5bf9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.282981 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:41:37.285948 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.287346 14019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:37.287499 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "d68237e0b86e4753a21d66fa2c66e362"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-xltq"
server_key: "fe3f8663c6c97d45ce9250de15c8b550"
server_key_iv: "f7187d86263770ad83899243b23b5bf9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.287621 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:37.305963 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:37.309485 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:37.309676 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:37.316439 14019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37833
I20260927 00:41:37.316447 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37833 every 8 connection(s)
I20260927 00:41:37.317457 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:37.318660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14019
I20260927 00:41:37.318778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:37.319048 392 external_mini_cluster.cc:1428] Setting key d415ac49ece3576fe4b87af43fe29f7a
I20260927 00:41:37.320658 14087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:37.327349 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362: Bootstrap starting.
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:37.330039 14087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:37.330965 14087 log.cc:824] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:37.333240 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362: No bootstrap required, opened a new log
I20260927 00:41:37.334110 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.320287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.0.98.62:37833)
0927 00:41:37.320773 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:37.320792 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:37.320835 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:41:37.321660 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:37.321685 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:37.322133 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:37.322501 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:37.323214 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.324713 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.325598 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.326316 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.326473 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.329160 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:37.329166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:37.329182 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:37.329220 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:37.331147 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.331575 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.331581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.331587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.331724 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.332026 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.332028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.332029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.332543 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:37.332695 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:37.332982 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 00:41:37.333265 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260927 00:41:37.337148 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:37.337260 14087 raft_consensus.cc:348] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } }
I20260927 00:41:37.337503 14087 raft_consensus.cc:374] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:37.337568 14087 raft_consensus.cc:729] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68237e0b86e4753a21d66fa2c66e362, State: Initialized, Role: FOLLOWER
I20260927 00:41:37.337973 14087 consensus_queue.cc:260] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } }
I20260927 00:41:37.338102 14087 raft_consensus.cc:388] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:37.338169 14087 raft_consensus.cc:482] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:37.338268 14087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:37.339270 14087 raft_consensus.cc:504] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } }
I20260927 00:41:37.339610 14087 leader_election.cc:302] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d68237e0b86e4753a21d66fa2c66e362; no voters:
I20260927 00:41:37.339920 14087 leader_election.cc:288] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:37.340044 14092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:37.340319 14092 raft_consensus.cc:686] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [term 1 LEADER]: Becoming Leader. State: Replica: d68237e0b86e4753a21d66fa2c66e362, State: Running, Role: LEADER
I20260927 00:41:37.340626 14092 consensus_queue.cc:237] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } }
I20260927 00:41:37.341220 14087 sys_catalog.cc:564] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:37.342396 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d68237e0b86e4753a21d66fa2c66e362. Latest consensus state: current_term: 1 leader_uuid: "d68237e0b86e4753a21d66fa2c66e362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } }
I20260927 00:41:37.342537 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:37.342835 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d68237e0b86e4753a21d66fa2c66e362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d68237e0b86e4753a21d66fa2c66e362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37833 } } }
I20260927 00:41:37.342954 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:37.342962 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:37.345947 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:37.352104 14101 catalog_manager.cc:1347] Generated new cluster ID: 654d473dc76643b99ccb126897b5799e
I20260927 00:41:37.352228 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:37.365201 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:37.366279 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:37.373063 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d68237e0b86e4753a21d66fa2c66e362: Generated new TSK 0
I20260927 00:41:37.373842 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:37.393211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14019
2026-09-27T00:41:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1358 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14114](info): set up 2 sockets
Sep 27 00:41:37 dist-test-slave-xltq krb5kdc[14114](info): commencing operation
krb5kdc: starting...
W20260927 00:41:38.364456 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:38 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:38.491598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:38.606420 14127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:38.606729 14127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:38.606772 14127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:38.611095 14127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:38.611184 14127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:38.611212 14127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:38.611234 14127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:38.611255 14127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:38.616982 14127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:38.618168 14127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:38.619242 14127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:38.628301 14133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:38.628476 14127 server_base.cc:1034] running on GCE node
W20260927 00:41:38.628405 14132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:38.628748 14135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:38.629273 14127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:38.630445 14127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:38.631625 14127 hybrid_clock.cc:648] HybridClock initialized: now 1790469698631597 us; error 41 us; skew 500 ppm
Sep 27 00:41:38 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:38.634457 14127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:38.635354 14127 webserver.cc:460] Webserver started at http://127.0.98.62:39455/ using document root <none> and password file <none>
I20260927 00:41:38.635895 14127 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:38.635962 14127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:38.636162 14127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:38.638332 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "ba15196dedfb4cdc8c9f305a25bc5a90"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "829f76f11f028ded8910d9bcbc648017"
server_key_iv: "68dc27e1175e124866ce791f1237b510"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.639039 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba15196dedfb4cdc8c9f305a25bc5a90"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "829f76f11f028ded8910d9bcbc648017"
server_key_iv: "68dc27e1175e124866ce791f1237b510"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.643062 14127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:41:38.645946 14142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:38.647444 14127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:38.647585 14127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ba15196dedfb4cdc8c9f305a25bc5a90"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "829f76f11f028ded8910d9bcbc648017"
server_key_iv: "68dc27e1175e124866ce791f1237b510"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.647717 14127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:38.659670 14127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:38.663559 14127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:38.663771 14127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:38.671368 14127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35829
I20260927 00:41:38.671705 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35829 every 8 connection(s)
I20260927 00:41:38.672533 14127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:38.675581 14195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:38.678544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14127
I20260927 00:41:38.678690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:38.678975 392 external_mini_cluster.cc:1428] Setting key a8b55cdb3528a7c7a33af396964eaa3d
I20260927 00:41:38.682054 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap starting.
I20260927 00:41:38.684774 14195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:38.685761 14195 log.cc:824] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:38.688472 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: No bootstrap required, opened a new log
Sep 27 00:41:38 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:38.691700 14195 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:38.691969 14195 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:38.692014 14195 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Initialized, Role: FOLLOWER
I20260927 00:41:38.692574 14195 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:38.692767 14195 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:38.692843 14195 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:38.692945 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:38.694207 14195 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:38.694643 14195 leader_election.cc:302] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba15196dedfb4cdc8c9f305a25bc5a90; no voters:
I20260927 00:41:38.695012 14195 leader_election.cc:288] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:38.695283 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:38.680426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.0.98.62:35829)
0927 00:41:38.680913 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:38.680929 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:38.680969 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:41:38.681656 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:38.681674 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:38.682071 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:38.682408 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:38.683295 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.684487 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.685668 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.686304 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.686505 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:38.689399 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:38.689405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:38.689426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:38.689464 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:38.691541 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:38.692075 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:38.692083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:38.692089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:38.692282 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:38.692580 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:38.692583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:38.692585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:38.693198 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:38.693430 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:38.693754 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 00:41:38.694136 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":128,"threads_started":1}
I20260927 00:41:38.695348 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:38.695583 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 LEADER]: Becoming Leader. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Running, Role: LEADER
I20260927 00:41:38.695952 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:38.696436 14195 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:38.698100 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba15196dedfb4cdc8c9f305a25bc5a90. Latest consensus state: current_term: 1 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:38.698251 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:38.698530 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:38.698887 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
W20260927 00:41:38.701349 14215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:38.701438 14215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:38.701565 14214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:38.702587 14214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:38.709754 14214 catalog_manager.cc:1347] Generated new cluster ID: 36dbf44ebe9943f1bfb1a40838090469
I20260927 00:41:38.709846 14214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:38.735782 14214 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:38.736863 14214 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:38.743852 14214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Generated new TSK 0
I20260927 00:41:38.744886 14214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:38.784878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:38.899816 14221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:38.900117 14221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:38.900182 14221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:38.904644 14221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:38.904735 14221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:38.904800 14221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:38.910476 14221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:38.911702 14221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:38.912799 14221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:38.923084 14227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:38.923084 14226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:38.923108 14229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:38.923426 14221 server_base.cc:1034] running on GCE node
I20260927 00:41:38.923864 14221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:38.924556 14221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:38.925740 14221 hybrid_clock.cc:648] HybridClock initialized: now 1790469698925685 us; error 72 us; skew 500 ppm
Sep 27 00:41:38 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:38.928642 14221 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:38.929536 14221 webserver.cc:460] Webserver started at http://127.0.98.1:40377/ using document root <none> and password file <none>
I20260927 00:41:38.930083 14221 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:38.930140 14221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:38.930310 14221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:38.932461 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea25c9d62f834badafac6d614dc9805f"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "990e4fdebd357f0f90a30a283fe8956e"
server_key_iv: "6418922818a49edc1ab2285996b81150"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.933032 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea25c9d62f834badafac6d614dc9805f"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "990e4fdebd357f0f90a30a283fe8956e"
server_key_iv: "6418922818a49edc1ab2285996b81150"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.937059 14221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:41:38.939644 14236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:38.941134 14221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:38.941269 14221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "ea25c9d62f834badafac6d614dc9805f"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "990e4fdebd357f0f90a30a283fe8956e"
server_key_iv: "6418922818a49edc1ab2285996b81150"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.941396 14221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:38.967348 14221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:38.971100 14221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:38.971279 14221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:38.971935 14221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:38.972831 14221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:38.972877 14221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:38.972918 14221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:38.972954 14221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:38.982244 14221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41241
I20260927 00:41:38.982276 14349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41241 every 8 connection(s)
I20260927 00:41:38.983296 14221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:38.992425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14221
I20260927 00:41:38.992580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:38.992851 392 external_mini_cluster.cc:1428] Setting key b32465f4971f5525ba89200215c2bf44
Sep 27 00:41:38 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.000046 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:38.985609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42727 (local address 127.0.98.62:35829)
0927 00:41:38.985772 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:38.985777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:38.987078 (+ 1301us) server_negotiation.cc:407] Connection header received
0927 00:41:38.988081 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:38.988084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:38.988152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:38.988249 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:38.989967 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.990582 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.991981 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.992363 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.992490 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:38.995851 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:38.995856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:38.995858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:38.995886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:38.997649 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:38.998336 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:38.998340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:38.998341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:38.998456 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:38.998853 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:38.998857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:38.998858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:38.999060 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:38.999151 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:38.999730 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 00:41:38.999873 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:41:39.001060 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:38.986082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:42727)
0927 00:41:38.986888 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:38.986940 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:38.987820 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:38.988644 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:38.988654 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:38.989096 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:38.989647 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:38.989664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.990812 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.990815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:38.991825 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:38.991839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:38.992604 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:38.992610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:38.992816 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:38.993508 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:38.993526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:38.995638 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:38.997804 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:38.997812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:38.997834 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:38.998215 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:38.998574 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:38.998578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:38.998580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:38.998753 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:38.999169 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:38.999177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:38.999483 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 00:41:38.999664 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.000048 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":191,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:39.002722 14350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:39.003108 14350 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.003671 14350 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:39.005707 14158 ts_manager.cc:194] Registered new tserver with Master: ea25c9d62f834badafac6d614dc9805f (127.0.98.1:41241)
I20260927 00:41:39.006920 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:39.024255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.141911 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.142203 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.142248 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.146554 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.146631 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.146695 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:39.152351 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:39.153453 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.154457 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:41:39.164779 14355 server_base.cc:1034] running on GCE node
W20260927 00:41:39.164646 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.164726 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.164646 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.165436 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.166145 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.167331 14355 hybrid_clock.cc:648] HybridClock initialized: now 1790469699167314 us; error 43 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.170429 14355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:39.171466 14355 webserver.cc:460] Webserver started at http://127.0.98.2:41645/ using document root <none> and password file <none>
I20260927 00:41:39.172056 14355 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.172122 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.172356 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:39.174517 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "84fdfd237d1f4550acc3fa381ce3719e"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "70fa53e3c1ba796bb97d190832e9e1ef"
server_key_iv: "fdc3ed526e098988ea21abfe78e1d130"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.175079 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84fdfd237d1f4550acc3fa381ce3719e"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "70fa53e3c1ba796bb97d190832e9e1ef"
server_key_iv: "fdc3ed526e098988ea21abfe78e1d130"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.179344 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:41:39.182268 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.183826 14355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:39.183977 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "84fdfd237d1f4550acc3fa381ce3719e"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "70fa53e3c1ba796bb97d190832e9e1ef"
server_key_iv: "fdc3ed526e098988ea21abfe78e1d130"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.184113 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.200618 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.204068 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.204293 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.204921 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:39.205771 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:39.205828 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.205886 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:39.205911 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.215224 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46695
I20260927 00:41:39.215339 14483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46695 every 8 connection(s)
I20260927 00:41:39.216300 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:39.221323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14355
I20260927 00:41:39.221448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:39.221675 392 external_mini_cluster.cc:1428] Setting key 5ad079c9eb9053419357332218c3cbc5
Sep 27 00:41:39 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:39.232669 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.218375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43239 (local address 127.0.98.62:35829)
0927 00:41:39.218559 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.218564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.219665 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 00:41:39.220678 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.220681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.220738 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.220821 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.222306 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.223010 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.224389 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.224729 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.224859 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.228131 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.228136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.228139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.228176 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.230087 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.230720 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.230724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.230725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.230847 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.231248 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.231252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.231254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.231533 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.231642 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.232386 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.232537 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 00:41:39.233821 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.218759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:43239)
0927 00:41:39.219500 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.219546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.220430 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.221015 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.221040 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.221519 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:39.222140 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.222163 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.223178 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.223182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.224240 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.224255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.224982 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.224986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.225231 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.225990 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:39.226011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:39.227874 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:39.230259 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.230266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.230284 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.230595 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.230959 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.230961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.230963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.231133 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.231742 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:39.231749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:39.232040 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.232308 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.232820 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:39.235234 14484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:39.235505 14484 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.236088 14484 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
I20260927 00:41:39.237481 14158 ts_manager.cc:194] Registered new tserver with Master: 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
I20260927 00:41:39.238123 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:39.252153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.367914 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.368259 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.368317 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.372524 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.372605 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.372668 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:39.378304 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:39.379390 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.380467 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:39.391038 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.391119 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.391043 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.391331 14489 server_base.cc:1034] running on GCE node
I20260927 00:41:39.391912 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.392624 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.393816 14489 hybrid_clock.cc:648] HybridClock initialized: now 1790469699393785 us; error 47 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.397188 14489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:39.398308 14489 webserver.cc:460] Webserver started at http://127.0.98.3:46793/ using document root <none> and password file <none>
I20260927 00:41:39.398908 14489 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.398981 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.399165 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:39.401409 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b4b78a88ada461490ce4f69e19f6de7"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "07f6e994f5bac67f28b4fe8ade3cb414"
server_key_iv: "eae87b552f5f6b538a176f24beeb5e44"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.401973 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b4b78a88ada461490ce4f69e19f6de7"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "07f6e994f5bac67f28b4fe8ade3cb414"
server_key_iv: "eae87b552f5f6b538a176f24beeb5e44"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.405982 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:41:39.408759 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.410207 14489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:39.410321 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "0b4b78a88ada461490ce4f69e19f6de7"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "07f6e994f5bac67f28b4fe8ade3cb414"
server_key_iv: "eae87b552f5f6b538a176f24beeb5e44"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.410437 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.428687 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.432560 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.432740 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.433324 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:39.434175 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:39.434231 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.434271 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:39.434291 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.443356 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43369
I20260927 00:41:39.443391 14617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43369 every 8 connection(s)
I20260927 00:41:39.444324 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:39.449427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14489
I20260927 00:41:39.449530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:39.449780 392 external_mini_cluster.cc:1428] Setting key 2ddcc3bedf90ec55029ed4a0f4169e3e
Sep 27 00:41:39 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.458513 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.446185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51793 (local address 127.0.98.62:35829)
0927 00:41:39.446323 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.446327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.447432 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 00:41:39.448436 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.448440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.448497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.448580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.449775 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.450342 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.451514 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.451793 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.451883 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.454642 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.454645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.454647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.454673 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.456101 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.456805 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.456809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.456810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.456903 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.457352 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.457356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.457357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.457561 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.457645 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.458257 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.458381 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:41:39.459334 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.446554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:51793)
0927 00:41:39.447277 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.447321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.448138 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.448738 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.448745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.449106 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:39.449621 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.449641 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.450462 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.450466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.451398 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.451408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.451935 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.451939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.452120 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.452766 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:39.452782 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:39.454398 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:39.456267 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.456274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.456293 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.456623 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.457016 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.457019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.457020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.457209 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.457677 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:39.457683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:39.457967 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.458181 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.458481 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":126,"threads_started":1}
I20260927 00:41:39.460659 14618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:39.460916 14618 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.461524 14618 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
I20260927 00:41:39.462849 14159 ts_manager.cc:194] Registered new tserver with Master: 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369)
I20260927 00:41:39.463603 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51793
I20260927 00:41:39.463861 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:39.474088 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.465984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.0.98.62:35829)
0927 00:41:39.466114 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.466118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.466243 (+ 125us) server_negotiation.cc:407] Connection header received
0927 00:41:39.466344 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.466346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.466385 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.466450 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.467303 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.467889 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.468851 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.469063 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.469147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.470572 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.470575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.470577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.470603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.472134 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.472603 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.472606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.472607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.472736 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.473101 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.473103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.473104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.473361 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.473478 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.473795 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.473932 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:41:39.477582 14159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:39.481087 14159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:39.497598 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.0.1:51950)
0927 00:41:39.491944 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.491988 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.492122 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.493025 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.493031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.493069 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:39.493420 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.493427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.494451 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.494454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.495686 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.495697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.496921 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.496925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.497056 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497103 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.497269 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.497419 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":195,"spinlock_wait_cycles":5248,"thread_start_us":109,"threads_started":1}
I20260927 00:41:39.497776 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.492109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.3:43369)
0927 00:41:39.492515 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.492527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.492550 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:41:39.492628 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.492633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.492852 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.493567 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:39.493758 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.494925 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.496314 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497076 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497194 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497406 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.497462 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.497634 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":173,"threads_started":1}
I20260927 00:41:39.498135 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43369 (local address 127.0.0.1:54330)
0927 00:41:39.492295 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.492305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.492368 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.493026 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.493031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.493071 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:39.493277 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.493285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.495071 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.495075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.496129 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.496138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497206 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.497337 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497363 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.497428 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.497542 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":76,"spinlock_wait_cycles":6016,"thread_start_us":104,"threads_started":1}
I20260927 00:41:39.498304 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.492602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.0.98.1:41241)
0927 00:41:39.492908 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.492921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.492941 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:41:39.493018 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.493022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.493203 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.493335 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:39.493733 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.494925 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.496564 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497373 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497481 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497994 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.498052 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.498166 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
I20260927 00:41:39.498454 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.0.98.2:46695)
0927 00:41:39.492570 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.492578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.492597 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:41:39.492658 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.492661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.492847 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.493289 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:39.493508 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.494330 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.495829 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.496791 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.496897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497056 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.498152 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.498288 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":124,"threads_started":1}
I20260927 00:41:39.498610 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41241 (local address 127.0.0.1:58638)
0927 00:41:39.492243 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.492257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.492351 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.493490 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.493493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.493509 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:39.493605 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.493611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.495319 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.495322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.496423 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.496433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497621 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.497711 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.497734 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.497797 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.497862 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":114,"threads_started":1}
I20260927 00:41:39.501897 14552 tablet_service.cc:1467] Processing CreateTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 (DEFAULT_TABLE table=test-table [id=0c9b699bc31040e5b9fb51ce24844443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:39.501895 14284 tablet_service.cc:1467] Processing CreateTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 (DEFAULT_TABLE table=test-table [id=0c9b699bc31040e5b9fb51ce24844443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:39.502292 14418 tablet_service.cc:1467] Processing CreateTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 (DEFAULT_TABLE table=test-table [id=0c9b699bc31040e5b9fb51ce24844443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:39.502986 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fa5039559a4cbfb6193dc1c97967f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:39.503007 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fa5039559a4cbfb6193dc1c97967f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:39.503250 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fa5039559a4cbfb6193dc1c97967f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:39.510627 14635 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Bootstrap starting.
I20260927 00:41:39.511535 14637 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Bootstrap starting.
I20260927 00:41:39.512018 14636 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Bootstrap starting.
I20260927 00:41:39.513635 14635 tablet_bootstrap.cc:654] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:39.513954 14637 tablet_bootstrap.cc:654] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:39.514343 14636 tablet_bootstrap.cc:654] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:39.514644 14635 log.cc:824] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:39.514740 14637 log.cc:824] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:39.515581 14636 log.cc:824] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:39.517190 14637 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: No bootstrap required, opened a new log
I20260927 00:41:39.517227 14635 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: No bootstrap required, opened a new log
I20260927 00:41:39.517431 14635 ts_tablet_manager.cc:1397] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:41:39.517432 14637 ts_tablet_manager.cc:1397] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:39.517892 14636 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: No bootstrap required, opened a new log
I20260927 00:41:39.518079 14636 ts_tablet_manager.cc:1397] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:39.521306 14635 raft_consensus.cc:348] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.521595 14635 raft_consensus.cc:374] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:39.521654 14635 raft_consensus.cc:729] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Initialized, Role: FOLLOWER
I20260927 00:41:39.521508 14637 raft_consensus.cc:348] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.521831 14637 raft_consensus.cc:374] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:39.521896 14637 raft_consensus.cc:729] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fdfd237d1f4550acc3fa381ce3719e, State: Initialized, Role: FOLLOWER
I20260927 00:41:39.522190 14635 consensus_queue.cc:260] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.522317 14637 consensus_queue.cc:260] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.521924 14636 raft_consensus.cc:348] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.522934 14636 raft_consensus.cc:374] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:39.522931 14350 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
I20260927 00:41:39.522989 14636 raft_consensus.cc:729] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4b78a88ada461490ce4f69e19f6de7, State: Initialized, Role: FOLLOWER
I20260927 00:41:39.523461 14637 ts_tablet_manager.cc:1428] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:39.523454 14636 consensus_queue.cc:260] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.524016 14484 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
I20260927 00:41:39.524462 14636 ts_tablet_manager.cc:1428] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 00:41:39.525029 14618 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
I20260927 00:41:39.525811 14635 ts_tablet_manager.cc:1428] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:41:39.639070 14641 raft_consensus.cc:482] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:39.639318 14641 raft_consensus.cc:504] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.640520 14641 leader_election.cc:288] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695), 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369)
I20260927 00:41:39.644558 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.640921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37399 (local address 127.0.98.3:43369)
0927 00:41:39.641096 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.641101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.641327 (+ 226us) server_negotiation.cc:407] Connection header received
0927 00:41:39.641477 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.641480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.641526 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.641584 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:39.641888 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.642393 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.643566 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.643975 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.644070 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.644135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.644345 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.644429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 00:41:39.644582 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.640809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43369 (local address 127.0.98.1:37399)
0927 00:41:39.641229 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.641240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.641373 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.641620 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.641623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.641637 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:39.641789 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.641796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.642505 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.642508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.643437 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.643446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.644101 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.644105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.644224 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.644249 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.644321 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.644393 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":102,"spinlock_wait_cycles":7808,"thread_start_us":125,"threads_started":1}
I20260927 00:41:39.644605 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.640923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59165 (local address 127.0.98.2:46695)
0927 00:41:39.641053 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.641058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.641315 (+ 257us) server_negotiation.cc:407] Connection header received
0927 00:41:39.641481 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.641485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.641538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.641615 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:39.641908 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.642391 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.643568 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.643998 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.644098 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.644163 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.644418 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.644492 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:39.644989 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.640803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.98.1:59165)
0927 00:41:39.641211 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.641223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.641371 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.641634 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.641636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.641651 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:39.641804 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.641810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.642492 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.642496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.643441 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.643451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.644118 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.644122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.644278 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.644300 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.644773 (+ 473us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.644852 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":60,"spinlock_wait_cycles":44544,"thread_start_us":161,"threads_started":1}
I20260927 00:41:39.645376 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fa5039559a4cbfb6193dc1c97967f5" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4b78a88ada461490ce4f69e19f6de7" is_pre_election: true
I20260927 00:41:39.645674 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fa5039559a4cbfb6193dc1c97967f5" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84fdfd237d1f4550acc3fa381ce3719e" is_pre_election: true
I20260927 00:41:39.645742 14572 raft_consensus.cc:2454] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 0.
I20260927 00:41:39.645943 14438 raft_consensus.cc:2454] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 0.
I20260927 00:41:39.646397 14237 leader_election.cc:302] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4b78a88ada461490ce4f69e19f6de7, ea25c9d62f834badafac6d614dc9805f; no voters:
I20260927 00:41:39.646703 14641 raft_consensus.cc:2781] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:39.646787 14641 raft_consensus.cc:482] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:39.646818 14641 raft_consensus.cc:3037] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:39.647861 14641 raft_consensus.cc:504] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.648273 14641 leader_election.cc:288] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 election: Requested vote from peers 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695), 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369)
I20260927 00:41:39.648676 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fa5039559a4cbfb6193dc1c97967f5" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4b78a88ada461490ce4f69e19f6de7"
I20260927 00:41:39.648680 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fa5039559a4cbfb6193dc1c97967f5" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84fdfd237d1f4550acc3fa381ce3719e"
I20260927 00:41:39.648814 14438 raft_consensus.cc:3037] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:39.648813 14572 raft_consensus.cc:3037] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:39.650130 14438 raft_consensus.cc:2454] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 1.
I20260927 00:41:39.650130 14572 raft_consensus.cc:2454] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 1.
I20260927 00:41:39.650480 14239 leader_election.cc:302] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84fdfd237d1f4550acc3fa381ce3719e, ea25c9d62f834badafac6d614dc9805f; no voters:
I20260927 00:41:39.650657 14641 raft_consensus.cc:2781] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:39.650870 14641 raft_consensus.cc:686] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 LEADER]: Becoming Leader. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Running, Role: LEADER
I20260927 00:41:39.651232 14641 consensus_queue.cc:237] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:39.654748 14157 catalog_manager.cc:5495] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f reported cstate change: term changed from 0 to 1, leader changed from <none> to ea25c9d62f834badafac6d614dc9805f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea25c9d62f834badafac6d614dc9805f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:39.691694 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.687739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.0.98.1:41241)
0927 00:41:39.687895 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.687900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.687997 (+ 97us) server_negotiation.cc:407] Connection header received
0927 00:41:39.688264 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.688267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.688340 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.688458 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:39.688707 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.689280 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.690184 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.690427 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.690530 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.690774 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:39.691227 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:39.691331 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.691438 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.691524 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:41:39.695286 14619 tablet.cc:2369] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:39.698501 14438 raft_consensus.cc:1261] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Refusing update from remote peer ea25c9d62f834badafac6d614dc9805f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:39.698501 14572 raft_consensus.cc:1261] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Refusing update from remote peer ea25c9d62f834badafac6d614dc9805f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:39.699363 14641 consensus_queue.cc:1035] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:39.699576 14646 consensus_queue.cc:1035] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:39.707233 14649 mvcc.cc:205] Tried to move back new op lower bound from 7333763889957281792 to 7333763889774690304. Current Snapshot: MvccSnapshot[applied={T|T < 7333763889957281792}]
I20260927 00:41:39.707849 14651 mvcc.cc:205] Tried to move back new op lower bound from 7333763889957281792 to 7333763889774690304. Current Snapshot: MvccSnapshot[applied={T|T < 7333763889957281792}]
I20260927 00:41:39.709815 14650 mvcc.cc:205] Tried to move back new op lower bound from 7333763889957281792 to 7333763889774690304. Current Snapshot: MvccSnapshot[applied={T|T < 7333763889957281792}]
I20260927 00:41:39.717659 14158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:39.717831 14158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:39.720582 14158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Sending DeleteTablet for 3 replicas of tablet e6fa5039559a4cbfb6193dc1c97967f5
I20260927 00:41:39.721494 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54330
I20260927 00:41:39.721494 14284 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58638
I20260927 00:41:39.721508 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51950
I20260927 00:41:39.722110 14661 tablet_replica.cc:330] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: stopping tablet replica
I20260927 00:41:39.722409 14660 tablet_replica.cc:330] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: stopping tablet replica
I20260927 00:41:39.722512 14661 raft_consensus.cc:2229] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:39.722564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14221
I20260927 00:41:39.722683 14660 raft_consensus.cc:2229] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:39.722886 14661 raft_consensus.cc:2258] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:39.722903 14660 raft_consensus.cc:2258] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:39.724993 14661 ts_tablet_manager.cc:1905] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:39.725052 14660 ts_tablet_manager.cc:1905] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:39.727748 14660 ts_tablet_manager.cc:1918] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:39.727883 14660 log.cc:1196] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/e6fa5039559a4cbfb6193dc1c97967f5
I20260927 00:41:39.727918 14661 ts_tablet_manager.cc:1918] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:39.728044 14661 log.cc:1196] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/e6fa5039559a4cbfb6193dc1c97967f5
I20260927 00:41:39.728338 14660 ts_tablet_manager.cc:1939] T e6fa5039559a4cbfb6193dc1c97967f5 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting consensus metadata
I20260927 00:41:39.728374 14661 ts_tablet_manager.cc:1939] T e6fa5039559a4cbfb6193dc1c97967f5 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting consensus metadata
I20260927 00:41:39.729710 14143 catalog_manager.cc:4841] TS 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): tablet e6fa5039559a4cbfb6193dc1c97967f5 (table test-table [id=0c9b699bc31040e5b9fb51ce24844443]) successfully deleted
I20260927 00:41:39.729773 14145 catalog_manager.cc:4841] TS 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): tablet e6fa5039559a4cbfb6193dc1c97967f5 (table test-table [id=0c9b699bc31040e5b9fb51ce24844443]) successfully deleted
W20260927 00:41:39.730310 14144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41241 (error 108)
I20260927 00:41:39.731094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14355
I20260927 00:41:39.731745 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.731182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41241 (local address 127.0.0.1:58662)
0927 00:41:39.731596 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.731665 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
W20260927 00:41:39.732019 14144 catalog_manager.cc:4584] TS ea25c9d62f834badafac6d614dc9805f (127.0.98.1:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6fa5039559a4cbfb6193dc1c97967f5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41241: connect: Connection refused (error 111)
I20260927 00:41:39.736186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14489
I20260927 00:41:39.741633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14127
I20260927 00:41:39.747640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--webserver_interface=127.0.98.62
--webserver_port=39455
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.866694 14663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.867000 14663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.867043 14663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.871400 14663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:39.871500 14663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.871519 14663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.871536 14663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:39.871551 14663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:39.877395 14663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:39.878592 14663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.879683 14663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:39.893743 14669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.893743 14670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.893745 14672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.894387 14663 server_base.cc:1034] running on GCE node
I20260927 00:41:39.895076 14663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.896425 14663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.897642 14663 hybrid_clock.cc:648] HybridClock initialized: now 1790469699897623 us; error 40 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.901273 14663 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:39.902396 14663 webserver.cc:460] Webserver started at http://127.0.98.62:39455/ using document root <none> and password file <none>
I20260927 00:41:39.903131 14663 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.903234 14663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.908087 14663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:39.910926 14679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.912664 14663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:39.912827 14663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ba15196dedfb4cdc8c9f305a25bc5a90"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "829f76f11f028ded8910d9bcbc648017"
server_key_iv: "68dc27e1175e124866ce791f1237b510"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.913538 14663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.935716 14663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.939342 14663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.939558 14663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.947307 14663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35829
I20260927 00:41:39.947576 14731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35829 every 8 connection(s)
I20260927 00:41:39.948567 14663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:39.953444 14732 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:41:39.955134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14663
I20260927 00:41:39.955471 14732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap starting.
I20260927 00:41:39.955600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=40377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:39.973387 14732 log.cc:824] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:39.982839 14732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:39.983282 14732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap complete.
I20260927 00:41:39.988101 14732 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:39.988677 14732 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Initialized, Role: FOLLOWER
I20260927 00:41:39.989283 14732 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:39.989434 14732 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:39.989513 14732 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:39.989616 14732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:41:39.991892 14732 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:39.992352 14732 leader_election.cc:302] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba15196dedfb4cdc8c9f305a25bc5a90; no voters:
I20260927 00:41:39.992691 14732 leader_election.cc:288] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:41:39.992847 14736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:41:39.993120 14736 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 LEADER]: Becoming Leader. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Running, Role: LEADER
I20260927 00:41:39.993460 14736 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:39.994014 14732 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:39.995011 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:39.995175 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:39.995441 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba15196dedfb4cdc8c9f305a25bc5a90. Latest consensus state: current_term: 2 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:39.995519 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:39.996037 14745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:40.001901 14745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:40.002599 14745 catalog_manager.cc:1259] Loaded cluster ID: 36dbf44ebe9943f1bfb1a40838090469
I20260927 00:41:40.002665 14745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:40.004179 14745 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:41:40.008687 14745 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:41:40.014918 14745 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:40.015861 14745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Loaded TSK: 0
I20260927 00:41:40.017350 14745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:41:40.085633 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:40.085911 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:40.085963 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:40.090413 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:40.090492 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:40.090559 14734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:40.096517 14734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40377
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:40.097626 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:40.098764 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:40.109169 14759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.109184 14762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.109169 14760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.110069 14734 server_base.cc:1034] running on GCE node
I20260927 00:41:40.110594 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:40.111472 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:40.112684 14734 hybrid_clock.cc:648] HybridClock initialized: now 1790469700112648 us; error 46 us; skew 500 ppm
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:40.116825 14734 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:40.118230 14734 webserver.cc:460] Webserver started at http://127.0.98.1:40377/ using document root <none> and password file <none>
I20260927 00:41:40.118989 14734 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:40.119108 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:40.124871 14734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:41:40.127754 14769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.129557 14734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:40.129705 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "ea25c9d62f834badafac6d614dc9805f"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "990e4fdebd357f0f90a30a283fe8956e"
server_key_iv: "6418922818a49edc1ab2285996b81150"
server_key_version: "encryptionkey@0"
I20260927 00:41:40.130466 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:40.153229 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:40.157002 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:40.157215 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:40.157827 14734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:40.159049 14776 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:41:40.163419 14734 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:41:40.163558 14734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 00:41:40.163625 14734 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:41:40.165584 14734 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:41:40.165637 14734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:40.165987 14776 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Bootstrap starting.
I20260927 00:41:40.175134 14734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41241
I20260927 00:41:40.175226 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41241 every 8 connection(s)
I20260927 00:41:40.176136 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:40.183523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14734
I20260927 00:41:40.184044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46695
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=41645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:40.185169 14776 log.cc:824] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:40.194371 14776 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:40.194774 14776 tablet_bootstrap.cc:492] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Bootstrap complete.
I20260927 00:41:40.196054 14776 ts_tablet_manager.cc:1397] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260927 00:41:40.197881 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.178737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34655 (local address 127.0.98.62:35829)
0927 00:41:40.179330 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.179352 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.179481 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:41:40.180999 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.181023 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.181370 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.181679 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:40.183427 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.185139 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.186746 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.187259 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.187409 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.190601 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:40.190609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:40.190628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:40.190668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:40.193149 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.194040 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.194046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.194052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.194353 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.194822 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.194827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.194828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.195567 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:40.195697 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.196578 (+ 881us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.196926 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":205,"threads_started":1}
I20260927 00:41:40.197911 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.178663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:34655)
0927 00:41:40.179280 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.179352 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.180517 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.181828 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.181838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.182360 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:40.183185 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.183211 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.185379 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.185383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.186579 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.186592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.187430 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.187434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.187631 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.188328 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:40.188350 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:40.190304 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:40.193351 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.193360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.193377 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.193863 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.194483 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.194486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.194488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.194704 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.195706 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:40.195715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:40.196139 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.196424 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.196919 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":170,"threads_started":1}
I20260927 00:41:40.199220 14776 raft_consensus.cc:348] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:40.199663 14776 raft_consensus.cc:729] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Initialized, Role: FOLLOWER
I20260927 00:41:40.200160 14776 consensus_queue.cc:260] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } } peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } }
I20260927 00:41:40.201327 14776 ts_tablet_manager.cc:1428] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260927 00:41:40.201511 14884 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:40.201758 14884 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:40.202337 14884 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
I20260927 00:41:40.205171 14696 ts_manager.cc:194] Registered new tserver with Master: ea25c9d62f834badafac6d614dc9805f (127.0.98.1:41241)
I20260927 00:41:40.209347 14696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34655
I20260927 00:41:40.212528 14884 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:40.220659 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.208465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41241 (local address 127.0.0.1:58672)
0927 00:41:40.208947 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.209002 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.209169 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.209872 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.209879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.210348 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:40.210486 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.210495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.211954 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.211958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.212993 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.213007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.213703 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.213721 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.213860 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.214676 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:40.214691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:40.216414 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:40.218714 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.218724 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.218731 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.219020 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.219337 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.219339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.219340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.219421 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.219976 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:40.219981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:40.220107 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.220276 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.220515 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":149,"threads_started":1}
I20260927 00:41:40.220703 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.208541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.1:41241)
0927 00:41:40.209348 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.209361 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.209388 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:41:40.209470 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.209475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.209696 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.209846 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:40.210631 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.211806 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.213125 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.213574 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.213685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.216592 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:40.216604 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:40.216626 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:40.216660 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:40.218575 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.219143 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.219149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.219151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.219251 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.219495 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.219498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.219500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.219840 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:40.219985 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.220361 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.220535 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":703,"thread_start_us":77,"threads_started":1}
I20260927 00:41:40.221836 14818 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fa5039559a4cbfb6193dc1c97967f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58672
I20260927 00:41:40.222756 14893 tablet_replica.cc:330] stopping tablet replica
I20260927 00:41:40.223060 14893 raft_consensus.cc:2229] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:40.223306 14893 raft_consensus.cc:2258] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:40.224433 14893 ts_tablet_manager.cc:1905] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:40.228441 14893 ts_tablet_manager.cc:1918] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:40.228556 14893 log.cc:1196] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/e6fa5039559a4cbfb6193dc1c97967f5
I20260927 00:41:40.228782 14893 ts_tablet_manager.cc:1939] T e6fa5039559a4cbfb6193dc1c97967f5 P ea25c9d62f834badafac6d614dc9805f: Deleting consensus metadata
I20260927 00:41:40.230808 14681 catalog_manager.cc:4841] TS ea25c9d62f834badafac6d614dc9805f (127.0.98.1:41241): tablet e6fa5039559a4cbfb6193dc1c97967f5 (table test-table [id=0c9b699bc31040e5b9fb51ce24844443]) successfully deleted
W20260927 00:41:40.311149 14888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:40.311527 14888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:40.311594 14888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:40.315893 14888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:40.315973 14888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:40.316061 14888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:40.321650 14888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41645
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:40.322764 14888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:40.323803 14888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:40.333746 14901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.333874 14888 server_base.cc:1034] running on GCE node
W20260927 00:41:40.333690 14899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.333690 14898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.334481 14888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:40.335139 14888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:40.336331 14888 hybrid_clock.cc:648] HybridClock initialized: now 1790469700336241 us; error 49 us; skew 500 ppm
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:40.339020 14888 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:40.340000 14888 webserver.cc:460] Webserver started at http://127.0.98.2:41645/ using document root <none> and password file <none>
I20260927 00:41:40.340597 14888 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:40.340683 14888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:40.344952 14888 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:40.347255 14908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.348685 14888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:40.348804 14888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "84fdfd237d1f4550acc3fa381ce3719e"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "70fa53e3c1ba796bb97d190832e9e1ef"
server_key_iv: "fdc3ed526e098988ea21abfe78e1d130"
server_key_version: "encryptionkey@0"
I20260927 00:41:40.349440 14888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:40.361250 14888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:40.364578 14888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:40.364763 14888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:40.365408 14888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:40.366246 14888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:40.366294 14888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.366349 14888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:40.366371 14888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.375365 14888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46695
I20260927 00:41:40.375442 15021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46695 every 8 connection(s)
I20260927 00:41:40.376490 14888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:40.383555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 14888
I20260927 00:41:40.383970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43369
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=46793
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:40.391310 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.378420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47021 (local address 127.0.98.62:35829)
0927 00:41:40.378555 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.378559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.379591 (+ 1032us) server_negotiation.cc:407] Connection header received
0927 00:41:40.380608 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.380611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.380665 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.380779 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:40.382231 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.382780 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.384084 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.384495 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.384601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.387510 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:40.387513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:40.387515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:40.387546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:40.389146 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.389748 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.389752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.389754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.389835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.390159 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.390163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.390164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.390364 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:40.390449 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.390985 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.391128 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:41:40.392180 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.378780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:47021)
0927 00:41:40.379428 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.379470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.380344 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.381081 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.381089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.381490 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:40.382042 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.382058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.382924 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.382927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.383948 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.383960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.384667 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.384672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.384899 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.385642 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:40.385664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:40.387314 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:40.389341 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.389348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.389367 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.389646 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.389918 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.389921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.389923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.390071 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.390463 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:40.390470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:40.390751 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.390927 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.391291 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":108,"threads_started":1}
I20260927 00:41:40.393518 15022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:40.393790 15022 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:40.394326 15022 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
I20260927 00:41:40.395447 14696 ts_manager.cc:194] Registered new tserver with Master: 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
I20260927 00:41:40.396288 14696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47021
W20260927 00:41:40.500015 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:40.500351 15025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:40.500401 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:40.504645 15025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:40.504729 15025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:40.504791 15025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:40.510600 15025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46793
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:40.511711 15025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:40.512826 15025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:40.523132 15030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.523134 15033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.523126 15031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.523521 15025 server_base.cc:1034] running on GCE node
I20260927 00:41:40.524076 15025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:40.524827 15025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:40.526019 15025 hybrid_clock.cc:648] HybridClock initialized: now 1790469700526005 us; error 28 us; skew 500 ppm
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:40.528959 15025 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:40.530119 15025 webserver.cc:460] Webserver started at http://127.0.98.3:46793/ using document root <none> and password file <none>
I20260927 00:41:40.530663 15025 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:40.530750 15025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:40.534746 15025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:40.537021 15040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.538451 15025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:40.538587 15025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "0b4b78a88ada461490ce4f69e19f6de7"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "07f6e994f5bac67f28b4fe8ade3cb414"
server_key_iv: "eae87b552f5f6b538a176f24beeb5e44"
server_key_version: "encryptionkey@0"
I20260927 00:41:40.539196 15025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:40.554239 15025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:40.557605 15025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:40.557809 15025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:40.558399 15025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:40.559226 15025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:40.559273 15025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.559338 15025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:40.559362 15025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.568749 15025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43369
I20260927 00:41:40.568796 15153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43369 every 8 connection(s)
I20260927 00:41:40.569864 15025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:40.570978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15025
I20260927 00:41:40.579573 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.571663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.0.98.62:35829)
0927 00:41:40.571811 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.571816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.572078 (+ 262us) server_negotiation.cc:407] Connection header received
0927 00:41:40.572193 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.572229 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.572284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.572376 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:40.573098 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.573684 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.574649 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.574874 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.574961 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.576676 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:40.576679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:40.576681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:40.576712 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:40.578059 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.578458 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.578462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.578464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.578540 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.578776 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.578779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.578781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.578945 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:40.579131 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.579307 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.579418 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:40.584582 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.571707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46481 (local address 127.0.98.62:35829)
0927 00:41:40.571924 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.571927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.573308 (+ 1381us) server_negotiation.cc:407] Connection header received
0927 00:41:40.574204 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.574208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.574261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.574349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:40.575574 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.576448 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.577959 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.578207 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.578304 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.580715 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:40.580717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:40.580719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:40.580740 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:40.582139 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.582737 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.582740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.582741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.582827 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:40.583315 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:40.583319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:40.583321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:40.583523 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:40.583619 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.584239 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.584444 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260927 00:41:40.585494 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.572385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:46481)
0927 00:41:40.573140 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.573194 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.574007 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.574496 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.574504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.574876 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:40.575417 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.575449 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.576582 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.576586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.577831 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.577843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.578361 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.578366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.578533 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.579011 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:40.579027 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:40.580573 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:40.582301 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.582308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.582323 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.582604 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.582950 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:40.582953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:40.582954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:40.583118 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:40.583656 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:40.583662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:40.583948 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.584134 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.584582 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":133,"threads_started":1}
I20260927 00:41:40.586845 15154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35829
I20260927 00:41:40.587085 15154 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:40.587602 15154 heartbeater.cc:511] Master 127.0.98.62:35829 requested a full tablet report, sending...
I20260927 00:41:40.588738 14696 ts_manager.cc:194] Registered new tserver with Master: 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369)
I20260927 00:41:40.589308 14696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46481
I20260927 00:41:40.593484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:40.598064 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.594162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.0.98.62:35829)
0927 00:41:40.594404 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.594409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.594485 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:41:40.594652 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.594654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.594778 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.594875 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:40.595107 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.595689 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.596586 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.596815 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.596909 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.597243 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:40.597666 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:40.597777 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.597855 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.597924 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260927 00:41:40.598791 14696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:40.602108 14696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:40.610927 14818 tablet_service.cc:1467] Processing CreateTablet for tablet 997b672d089749c39ada7d86e28431c4 (DEFAULT_TABLE table=test-table [id=2f546750a6f24229ba375f3f651787c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:40.611691 14818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b672d089749c39ada7d86e28431c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:40.614857 14776 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Bootstrap starting.
I20260927 00:41:40.615549 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.609799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43369 (local address 127.0.0.1:54340)
0927 00:41:40.609921 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.609938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.610021 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.611172 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.611175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.611194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:40.611311 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.611318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.612805 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.612808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.614012 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.614024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.615074 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.615077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.615171 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.615228 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.615306 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.615408 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:41:40.615613 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.610003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.98.3:43369)
0927 00:41:40.610696 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.610710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.610735 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:41:40.610813 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.610817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.611007 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.611180 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:40.611494 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.612635 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.614153 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.614935 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.615067 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.615237 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.615356 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.615465 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":168,"threads_started":1}
I20260927 00:41:40.615850 14776 tablet_bootstrap.cc:654] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:40.617233 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.610528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.0.98.2:46695)
0927 00:41:40.611529 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.611541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.611562 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:40.611639 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.611643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.611824 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.611983 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:40.612238 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.613371 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.614470 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.616652 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.616762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.616916 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.616990 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.617081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":125,"threads_started":1}
I20260927 00:41:40.617230 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.610276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.0.1:51954)
0927 00:41:40.610541 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.610555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.610648 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.611967 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.611971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.611990 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:40.612092 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.612099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.613507 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.613510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.614329 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.614340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.616776 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.616779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.616869 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.616891 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.616958 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.617037 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
I20260927 00:41:40.617421 14776 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: No bootstrap required, opened a new log
I20260927 00:41:40.617522 14776 ts_tablet_manager.cc:1397] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:40.617972 14776 raft_consensus.cc:348] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.618108 14776 raft_consensus.cc:374] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:40.618153 14776 raft_consensus.cc:729] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Initialized, Role: FOLLOWER
I20260927 00:41:40.618280 14776 consensus_queue.cc:260] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.618530 14776 ts_tablet_manager.cc:1428] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:40.619192 15088 tablet_service.cc:1467] Processing CreateTablet for tablet 997b672d089749c39ada7d86e28431c4 (DEFAULT_TABLE table=test-table [id=2f546750a6f24229ba375f3f651787c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:40.620285 15088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b672d089749c39ada7d86e28431c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:40.626202 14956 tablet_service.cc:1467] Processing CreateTablet for tablet 997b672d089749c39ada7d86e28431c4 (DEFAULT_TABLE table=test-table [id=2f546750a6f24229ba375f3f651787c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:40.627485 14956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b672d089749c39ada7d86e28431c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:40.628798 15167 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Bootstrap starting.
I20260927 00:41:40.630784 15167 tablet_bootstrap.cc:654] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:40.631872 15167 log.cc:824] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:40.633925 15167 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: No bootstrap required, opened a new log
I20260927 00:41:40.634145 15167 ts_tablet_manager.cc:1397] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 00:41:40.634949 15169 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Bootstrap starting.
I20260927 00:41:40.637890 15169 tablet_bootstrap.cc:654] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:40.638370 15167 raft_consensus.cc:348] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.638695 15167 raft_consensus.cc:374] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:40.638746 15167 raft_consensus.cc:729] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4b78a88ada461490ce4f69e19f6de7, State: Initialized, Role: FOLLOWER
I20260927 00:41:40.638828 15169 log.cc:824] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:40.639247 15167 consensus_queue.cc:260] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.639988 15154 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
I20260927 00:41:40.640327 15167 ts_tablet_manager.cc:1428] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260927 00:41:40.641132 15169 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: No bootstrap required, opened a new log
I20260927 00:41:40.641336 15169 ts_tablet_manager.cc:1397] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:41:40.644474 15169 raft_consensus.cc:348] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.644731 15169 raft_consensus.cc:374] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:40.644767 15169 raft_consensus.cc:729] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fdfd237d1f4550acc3fa381ce3719e, State: Initialized, Role: FOLLOWER
I20260927 00:41:40.645170 15169 consensus_queue.cc:260] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.645826 15022 heartbeater.cc:503] Master 127.0.98.62:35829 was elected leader, sending a full tablet report...
I20260927 00:41:40.646136 15169 ts_tablet_manager.cc:1428] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 00:41:40.677399 14885 tablet.cc:2369] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:40.744346 14890 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:40.744709 14890 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.745939 14890 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
I20260927 00:41:40.750164 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.746242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.98.1:59449)
0927 00:41:40.746566 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.746580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.746714 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.747001 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.747004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.747022 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:40.747141 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.747148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.747897 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.747900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.748915 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.748927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.749637 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.749732 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.749800 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.749912 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.749990 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":54,"thread_start_us":103,"threads_started":1}
I20260927 00:41:40.750267 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.746389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49193 (local address 127.0.98.3:43369)
0927 00:41:40.746579 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.746583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.746784 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:41:40.746918 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.746920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.746971 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.747043 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:40.747267 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.747769 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749071 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.749703 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749805 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.749864 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.750094 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.750166 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 00:41:40.750300 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.746359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59449 (local address 127.0.98.2:46695)
0927 00:41:40.746495 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.746499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.746679 (+ 180us) server_negotiation.cc:407] Connection header received
0927 00:41:40.746843 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.746847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.746903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.746971 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:40.747251 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.747775 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749051 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.749503 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749589 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.749655 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.749952 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.750039 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:41:40.750785 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.746302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43369 (local address 127.0.98.1:49193)
0927 00:41:40.746686 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.746700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:40.746819 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:40.747071 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:40.747074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:40.747089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:40.747167 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.747172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.747888 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.747892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.748933 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:40.748945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.749834 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.749838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:40.749957 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.749983 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:41:40.750066 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:41:40.750140 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":69,"thread_start_us":103,"threads_started":1}
I20260927 00:41:40.751030 14976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b672d089749c39ada7d86e28431c4" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84fdfd237d1f4550acc3fa381ce3719e" is_pre_election: true
I20260927 00:41:40.751399 14976 raft_consensus.cc:2454] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 0.
I20260927 00:41:40.751395 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b672d089749c39ada7d86e28431c4" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4b78a88ada461490ce4f69e19f6de7" is_pre_election: true
I20260927 00:41:40.751696 15108 raft_consensus.cc:2454] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 0.
I20260927 00:41:40.751874 14772 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84fdfd237d1f4550acc3fa381ce3719e, ea25c9d62f834badafac6d614dc9805f; no voters:
I20260927 00:41:40.752177 14890 raft_consensus.cc:2781] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:40.752290 14890 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:40.752341 14890 raft_consensus.cc:3037] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:40.753397 14890 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.753800 14890 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 election: Requested vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
I20260927 00:41:40.754174 15108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b672d089749c39ada7d86e28431c4" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4b78a88ada461490ce4f69e19f6de7"
I20260927 00:41:40.754246 14976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b672d089749c39ada7d86e28431c4" candidate_uuid: "ea25c9d62f834badafac6d614dc9805f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84fdfd237d1f4550acc3fa381ce3719e"
I20260927 00:41:40.754305 15108 raft_consensus.cc:3037] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:40.754380 14976 raft_consensus.cc:3037] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:40.755376 15108 raft_consensus.cc:2454] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 1.
I20260927 00:41:40.755496 14976 raft_consensus.cc:2454] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea25c9d62f834badafac6d614dc9805f in term 1.
I20260927 00:41:40.755663 14770 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4b78a88ada461490ce4f69e19f6de7, ea25c9d62f834badafac6d614dc9805f; no voters:
I20260927 00:41:40.755828 14890 raft_consensus.cc:2781] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:40.756021 14890 raft_consensus.cc:686] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 LEADER]: Becoming Leader. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Running, Role: LEADER
I20260927 00:41:40.756317 14890 consensus_queue.cc:237] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:40.759298 14695 catalog_manager.cc:5495] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f reported cstate change: term changed from 0 to 1, leader changed from <none> to ea25c9d62f834badafac6d614dc9805f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ea25c9d62f834badafac6d614dc9805f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:40.773203 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.769580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.1:41241)
0927 00:41:40.769904 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:40.769913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:40.769925 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:40.769999 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:40.770001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:40.770056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:40.770172 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:40.770392 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.770901 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.771730 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:40.771953 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:40.772039 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:40.772313 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:40.772765 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:40.772874 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:40.772956 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 00:41:40.773017 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":121,"threads_started":1}
I20260927 00:41:40.777437 14976 raft_consensus.cc:1261] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Refusing update from remote peer ea25c9d62f834badafac6d614dc9805f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:40.777452 15108 raft_consensus.cc:1261] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Refusing update from remote peer ea25c9d62f834badafac6d614dc9805f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:40.778245 15175 consensus_queue.cc:1035] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:40.778450 14890 consensus_queue.cc:1035] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:40.786213 15178 mvcc.cc:205] Tried to move back new op lower bound from 7333763894377918464 to 7333763894300651520. Current Snapshot: MvccSnapshot[applied={T|T < 7333763894377918464}]
I20260927 00:41:40.796245 14696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:40.796427 14696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:40.798965 14696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Sending DeleteTablet for 3 replicas of tablet 997b672d089749c39ada7d86e28431c4
I20260927 00:41:40.799760 14818 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b672d089749c39ada7d86e28431c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58672
I20260927 00:41:40.799839 14956 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b672d089749c39ada7d86e28431c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51954
I20260927 00:41:40.799826 15088 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b672d089749c39ada7d86e28431c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54340
I20260927 00:41:40.800408 15188 tablet_replica.cc:330] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: stopping tablet replica
I20260927 00:41:40.800544 15188 raft_consensus.cc:2229] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:40.800635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14734
I20260927 00:41:40.800585 15190 tablet_replica.cc:330] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: stopping tablet replica
I20260927 00:41:40.802248 15190 raft_consensus.cc:2229] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:40.802552 15190 raft_consensus.cc:2258] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:40.802918 15189 tablet_replica.cc:330] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: stopping tablet replica
I20260927 00:41:40.803222 15189 raft_consensus.cc:2229] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:40.803520 15189 raft_consensus.cc:2258] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:40.804251 15190 ts_tablet_manager.cc:1905] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:40.804888 15189 ts_tablet_manager.cc:1905] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:41:40.806496 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41241 (error 108)
I20260927 00:41:40.806790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14888
I20260927 00:41:40.807729 15190 ts_tablet_manager.cc:1918] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:40.807828 15190 log.cc:1196] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/997b672d089749c39ada7d86e28431c4
I20260927 00:41:40.808128 15190 ts_tablet_manager.cc:1939] T 997b672d089749c39ada7d86e28431c4 P 0b4b78a88ada461490ce4f69e19f6de7: Deleting consensus metadata
I20260927 00:41:40.808559 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.808260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41241 (local address 127.0.0.1:58694)
0927 00:41:40.808392 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.808470 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:41:40.808841 14681 catalog_manager.cc:4584] TS ea25c9d62f834badafac6d614dc9805f (127.0.98.1:41241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 997b672d089749c39ada7d86e28431c4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41241: connect: Connection refused (error 111)
I20260927 00:41:40.809423 14680 catalog_manager.cc:4841] TS 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): tablet 997b672d089749c39ada7d86e28431c4 (table test-table [id=2f546750a6f24229ba375f3f651787c8]) successfully deleted
I20260927 00:41:40.812470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15025
I20260927 00:41:40.813848 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:40.812704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.0.1:51956)
0927 00:41:40.813699 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:40.813758 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899}
I20260927 00:41:40.817427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 14663
I20260927 00:41:40.823055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--webserver_interface=127.0.98.62
--webserver_port=39455
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:40.938539 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:40.938848 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:40.938890 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:40.943092 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:40.943181 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:40.943199 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:40.943215 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:40.943238 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:40.949379 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:40.950570 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:40.951668 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:40.960428 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:40.960448 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.960486 15192 server_base.cc:1034] running on GCE node
W20260927 00:41:40.960492 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:40.961393 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:40.962527 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:40.963688 15192 hybrid_clock.cc:648] HybridClock initialized: now 1790469700963676 us; error 28 us; skew 500 ppm
Sep 27 00:41:40 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:40.966569 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:40.967516 15192 webserver.cc:460] Webserver started at http://127.0.98.62:39455/ using document root <none> and password file <none>
I20260927 00:41:40.968086 15192 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:40.968147 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:40.972296 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:40.974607 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:40.976017 15192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:40.976150 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ba15196dedfb4cdc8c9f305a25bc5a90"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "829f76f11f028ded8910d9bcbc648017"
server_key_iv: "68dc27e1175e124866ce791f1237b510"
server_key_version: "encryptionkey@0"
I20260927 00:41:40.976753 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:40.988762 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:40.992405 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:40.992609 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:40.999671 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35829
I20260927 00:41:40.999981 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35829 every 8 connection(s)
I20260927 00:41:41.000869 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:41.005421 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:41:41.007189 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap starting.
I20260927 00:41:41.010236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15192
I20260927 00:41:41.010655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=40377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:41.025152 15260 log.cc:824] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.037833 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:41.038264 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90: Bootstrap complete.
I20260927 00:41:41.042496 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:41.042958 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.043505 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:41.043633 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:41.043696 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:41.043797 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:41:41.045992 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:41.046379 15260 leader_election.cc:302] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba15196dedfb4cdc8c9f305a25bc5a90; no voters:
I20260927 00:41:41.046718 15260 leader_election.cc:288] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:41:41.046835 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:41:41.047027 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [term 3 LEADER]: Becoming Leader. State: Replica: ba15196dedfb4cdc8c9f305a25bc5a90, State: Running, Role: LEADER
I20260927 00:41:41.047358 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } }
I20260927 00:41:41.048252 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:41.048444 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba15196dedfb4cdc8c9f305a25bc5a90. Latest consensus state: current_term: 3 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:41.048579 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:41.048941 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:41.049075 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba15196dedfb4cdc8c9f305a25bc5a90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35829 } } }
I20260927 00:41:41.049263 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba15196dedfb4cdc8c9f305a25bc5a90 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:41.054540 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:41.055323 15268 catalog_manager.cc:1259] Loaded cluster ID: 36dbf44ebe9943f1bfb1a40838090469
I20260927 00:41:41.055480 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:41:41.056618 15268 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9bdf5c9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9bdf5f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9bdf5c4de google::LogMessage::Flush() at ??:0
@ 0x7fb9bdf5da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9c889ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb9c88a0d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9c889d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb9c88e1640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9c01ae3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9c01aec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb9c01b03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb9c019f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9c17e36db start_thread at ??:0
@ 0x7fb9bca5a61f clone at ??:0
W20260927 00:41:41.142437 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.142736 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.142799 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.147086 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.147161 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.147243 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:41.153017 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40377
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:41.154222 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.155284 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.165179 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.165367 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.165702 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.165824 15262 server_base.cc:1034] running on GCE node
I20260927 00:41:41.166294 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.167001 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.168174 15262 hybrid_clock.cc:648] HybridClock initialized: now 1790469701168154 us; error 36 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.171067 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:41.172031 15262 webserver.cc:460] Webserver started at http://127.0.98.1:40377/ using document root <none> and password file <none>
I20260927 00:41:41.172634 15262 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.172699 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.176839 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:41.179302 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.180801 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:41.180927 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "ea25c9d62f834badafac6d614dc9805f"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-xltq"
server_key: "990e4fdebd357f0f90a30a283fe8956e"
server_key_iv: "6418922818a49edc1ab2285996b81150"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.181536 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.196560 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.199823 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.200014 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.200697 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.201850 15318 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:41:41.207664 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:41:41.207769 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 00:41:41.207845 15262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:41:41.209784 15262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:41:41.209842 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:41.210237 15318 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Bootstrap starting.
I20260927 00:41:41.220078 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41241
I20260927 00:41:41.220106 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41241 every 8 connection(s)
I20260927 00:41:41.221153 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:41.228000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15262
I20260927 00:41:41.228499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46695
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=41645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:41.232478 15318 log.cc:824] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.242520 15318 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:41.243005 15318 tablet_bootstrap.cc:492] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Bootstrap complete.
I20260927 00:41:41.244335 15318 ts_tablet_manager.cc:1397] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.012s
I20260927 00:41:41.247828 15318 raft_consensus.cc:348] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:41.248492 15318 raft_consensus.cc:729] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea25c9d62f834badafac6d614dc9805f, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.249100 15318 consensus_queue.cc:260] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:41.250253 15318 ts_tablet_manager.cc:1428] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 00:41:41.297190 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:41.223329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:60071)
0927 00:41:41.223874 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.223934 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.224856 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.296169 (+ 71313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
W20260927 00:41:41.297770 15426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35829 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35829: Client connection negotiation failed: client connection to 127.0.98.62:35829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:41:41.298604 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.298274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:37449)
0927 00:41:41.298428 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.298524 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:41:41.299396 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.299153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:45547)
0927 00:41:41.299270 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.299326 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 00:41:41.299621 15426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:41:41.346385 15429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.346691 15429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.346738 15429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.351097 15429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.351187 15429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.351251 15429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:41.357112 15429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41645
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:41.358227 15429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.359261 15429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.369369 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.369379 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.369481 15429 server_base.cc:1034] running on GCE node
W20260927 00:41:41.369413 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.370020 15429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.370668 15429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.371871 15429 hybrid_clock.cc:648] HybridClock initialized: now 1790469701371827 us; error 62 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.374990 15429 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:41.375980 15429 webserver.cc:460] Webserver started at http://127.0.98.2:41645/ using document root <none> and password file <none>
I20260927 00:41:41.376585 15429 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.376659 15429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.380897 15429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:41.383549 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.385089 15429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:41.385226 15429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "84fdfd237d1f4550acc3fa381ce3719e"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "70fa53e3c1ba796bb97d190832e9e1ef"
server_key_iv: "fdc3ed526e098988ea21abfe78e1d130"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.385848 15429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.400708 15429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.403743 15429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.403910 15429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.404541 15429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.405637 15453 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 00:41:41.409557 15453 ts_tablet_manager.cc:1852] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 00:41:41.409955 15453 ts_tablet_manager.cc:1905] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:41.414291 15453 ts_tablet_manager.cc:1918] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:41.414408 15453 log.cc:1196] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/997b672d089749c39ada7d86e28431c4
I20260927 00:41:41.414742 15453 ts_tablet_manager.cc:1939] T 997b672d089749c39ada7d86e28431c4 P 84fdfd237d1f4550acc3fa381ce3719e: Deleting consensus metadata
I20260927 00:41:41.415203 15429 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 00:41:41.415261 15429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260927 00:41:41.415338 15429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:41.415364 15429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.424875 15429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46695
I20260927 00:41:41.424966 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46695 every 8 connection(s)
I20260927 00:41:41.425904 15429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:41.430035 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.428437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:60283)
0927 00:41:41.429041 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.429178 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":131,"threads_started":1}
W20260927 00:41:41.430640 15561 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35829 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35829: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
I20260927 00:41:41.431597 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.431262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:37133)
0927 00:41:41.431430 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.431495 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 00:41:41.432610 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.432332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:55277)
0927 00:41:41.432463 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.432544 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 00:41:41.432838 15561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:41:41.435623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15429
I20260927 00:41:41.436024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43369
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35829
--webserver_port=46793
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:41.551124 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.551430 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.551474 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.555727 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.555804 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.555876 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:41.561340 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46793
--tserver_master_addrs=127.0.98.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:41.562512 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.563506 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.574091 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.574304 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.574095 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.574394 15564 server_base.cc:1034] running on GCE node
I20260927 00:41:41.574819 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.575495 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.576655 15564 hybrid_clock.cc:648] HybridClock initialized: now 1790469701576642 us; error 29 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-xltq krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.579631 15564 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:41.580778 15564 webserver.cc:460] Webserver started at http://127.0.98.3:46793/ using document root <none> and password file <none>
I20260927 00:41:41.581362 15564 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.581480 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.586082 15564 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:41.588625 15579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.590247 15564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:41.590368 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "0b4b78a88ada461490ce4f69e19f6de7"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-xltq"
server_key: "07f6e994f5bac67f28b4fe8ade3cb414"
server_key_iv: "eae87b552f5f6b538a176f24beeb5e44"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.590967 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.604703 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.608062 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.608283 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.608894 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.609722 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:41.609769 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.609808 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:41.609838 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.618947 15564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43369
I20260927 00:41:41.619028 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43369 every 8 connection(s)
I20260927 00:41:41.620019 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:41.622938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15564
I20260927 00:41:41.624439 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.622341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:37497)
0927 00:41:41.623301 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.623480 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":126,"threads_started":1}
W20260927 00:41:41.624991 15693 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35829 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35829: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
I20260927 00:41:41.625684 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.625436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:46105)
0927 00:41:41.625560 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.625620 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:41:41.626386 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.626163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:40105)
0927 00:41:41.626289 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.626334 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 00:41:41.626600 15693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:41:42.300789 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.300304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:40887)
0927 00:41:42.300548 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.300600 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260927 00:41:42.434028 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.433449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:36107)
0927 00:41:42.433817 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.433893 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":85,"threads_started":1}
I20260927 00:41:42.627810 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.627270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:44503)
0927 00:41:42.627609 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.627686 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260927 00:41:42.753204 15700 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:42.753425 15700 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:42.754482 15700 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
Sep 27 00:41:42 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:41:42 dist-test-slave-xltq krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:41:42.771200 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.754920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57229 (local address 127.0.98.2:46695)
0927 00:41:42.755248 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.755264 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.755306 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:41:42.756241 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.756248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.756702 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.757039 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:42.758805 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.760494 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.762486 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.763244 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.763425 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.766557 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:42.766563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:42.766574 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:42.766616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:42.768536 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769080 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:42.769086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:42.769092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:42.769214 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769528 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:42.769532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:42.769534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:42.770033 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:42.770152 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.770736 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.770998 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260927 00:41:42.771188 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.754749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46695 (local address 127.0.98.1:57229)
0927 00:41:42.755130 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.755143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.755241 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.757185 (+ 1944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.757192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.757790 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:42.758589 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.758612 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.760699 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.760703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.762335 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.762350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.763696 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.763700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.763791 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.764390 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:42.764401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:42.766301 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:42.768637 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:42.768640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:42.768643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:42.768970 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:42.769311 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:42.769315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:42.769445 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:42.770158 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:42.770165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:42.770431 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.770604 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.770985 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":192,"spinlock_wait_cycles":5248,"thread_start_us":77,"threads_started":1}
I20260927 00:41:42.771206 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.754681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43369 (local address 127.0.98.1:54081)
0927 00:41:42.754866 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.754880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.754984 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.757185 (+ 2201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.757192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.757790 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:42.758589 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.758612 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.760699 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.760703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.762335 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.762350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.763401 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.763406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.763627 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.764274 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:42.764297 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:42.766028 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:42.768601 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:42.768607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:42.768619 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:42.768975 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:42.769348 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:42.769353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:42.769456 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:42.770176 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:42.770179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:42.770431 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.770604 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.771057 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":449,"spinlock_wait_cycles":4736}
I20260927 00:41:42.771205 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.754912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54081 (local address 127.0.98.3:43369)
0927 00:41:42.755323 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.755343 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.755370 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:41:42.756187 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.756222 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.756702 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.757038 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:42.758805 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.760494 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.762486 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.763244 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.763425 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.766273 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:42.766280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:42.766299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:42.766333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:42.768392 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769106 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:42.769111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:42.769115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:42.769234 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:42.769558 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:42.769561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:42.769562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:42.770063 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:42.770175 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.770737 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.771015 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
W20260927 00:41:42.774175 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:42.774616 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:42.774691 15312 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:42.774958 15700 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:43.302058 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.301602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:42077)
0927 00:41:43.301889 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.301954 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260927 00:41:43.435755 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.435033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:56453)
0927 00:41:43.435544 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.435639 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":169,"threads_started":1}
I20260927 00:41:43.629438 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.628762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:46101)
0927 00:41:43.629197 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.629298 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":92,"threads_started":1}
I20260927 00:41:44.303583 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.303080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:49127)
0927 00:41:44.303397 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.303470 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260927 00:41:44.437508 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.436860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:59829)
0927 00:41:44.437325 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.437409 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260927 00:41:44.631000 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.630491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:40765)
0927 00:41:44.630812 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.630879 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":112,"threads_started":1}
I20260927 00:41:44.824074 15710 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:44.824352 15710 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:44.824936 15710 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
W20260927 00:41:44.826218 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:44.826437 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:44.826547 15312 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:44.826802 15710 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:45.305455 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:45.304847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:40347)
0927 00:41:45.305244 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:45.305334 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
I20260927 00:41:45.439318 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:45.438666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:53517)
0927 00:41:45.439117 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:45.439209 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
I20260927 00:41:45.632427 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:45.631874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:47011)
0927 00:41:45.632171 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:45.632292 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260927 00:41:46.307647 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.306958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:49685)
0927 00:41:46.307416 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:46.307512 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":99,"threads_started":1}
I20260927 00:41:46.441045 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.440553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:39035)
0927 00:41:46.440865 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:46.440930 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
I20260927 00:41:46.633905 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.633469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:45593)
0927 00:41:46.633741 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:46.633807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260927 00:41:47.238214 15717 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:47.238373 15717 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:47.238839 15717 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
W20260927 00:41:47.239692 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:47.239833 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:47.239913 15314 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:47.240061 15717 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:47.309114 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.308657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:39763)
0927 00:41:47.308944 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:47.309007 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260927 00:41:47.442353 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.441951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:53067)
0927 00:41:47.442197 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:47.442259 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260927 00:41:47.635108 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.634708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:37731)
0927 00:41:47.634948 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:47.635006 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260927 00:41:48.310539 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.309964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:47383)
0927 00:41:48.310375 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:48.310440 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":101,"threads_started":1}
I20260927 00:41:48.443596 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.443115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:56307)
0927 00:41:48.443432 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:48.443496 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":73,"threads_started":1}
I20260927 00:41:48.636298 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.635824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:44337)
0927 00:41:48.636112 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:48.636165 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260927 00:41:49.311793 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.311326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:60667)
0927 00:41:49.311618 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.311685 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260927 00:41:49.444932 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.444494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:42123)
0927 00:41:49.444763 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.444832 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260927 00:41:49.614645 15726 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:49.614853 15726 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:49.615406 15726 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
W20260927 00:41:49.616437 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:49.616585 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:49.616700 15314 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:49.616876 15726 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:49.637880 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.637262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:40435)
0927 00:41:49.637668 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.637764 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260927 00:41:50.313402 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.312742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:47311)
0927 00:41:50.313127 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.313224 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":79,"threads_started":1}
I20260927 00:41:50.446573 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.445946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:48369)
0927 00:41:50.446368 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.446456 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260927 00:41:50.639604 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.638983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:59057)
0927 00:41:50.639372 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.639463 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260927 00:41:51.314971 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.314450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:58661)
0927 00:41:51.314796 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.314861 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
I20260927 00:41:51.448576 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.447696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:47537)
0927 00:41:51.448346 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.448436 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":162,"threads_started":1}
I20260927 00:41:51.641455 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.640773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:38819)
0927 00:41:51.641211 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.641313 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
I20260927 00:41:52.316397 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.315782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:46251)
0927 00:41:52.316151 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.316262 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
I20260927 00:41:52.450237 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.449555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:34853)
0927 00:41:52.449992 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.450089 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
I20260927 00:41:52.643049 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.642465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:55551)
0927 00:41:52.642806 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.642870 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260927 00:41:53.317797 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.317329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:50577)
0927 00:41:53.317640 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.317697 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260927 00:41:53.451485 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.451029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:56423)
0927 00:41:53.451293 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.451383 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":69,"threads_started":1}
I20260927 00:41:53.644402 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.643924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:58105)
0927 00:41:53.644187 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.644275 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260927 00:41:54.319348 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.318652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:47789)
0927 00:41:54.319093 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.319203 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":80,"threads_started":1}
I20260927 00:41:54.433495 15741 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:54.433665 15741 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:54.434166 15741 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
W20260927 00:41:54.435325 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:54.435572 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:54.435652 15314 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:54.435878 15741 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:54.453022 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.452349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:42221)
0927 00:41:54.452781 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.452880 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":76,"threads_started":1}
I20260927 00:41:54.645991 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.645415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:35473)
0927 00:41:54.645786 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.645880 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260927 00:41:55.321085 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.320442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:35679)
0927 00:41:55.320877 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:55.320966 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":118,"threads_started":1}
I20260927 00:41:55.454761 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.454145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:51713)
0927 00:41:55.454511 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:55.454607 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
I20260927 00:41:55.647621 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.646943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.3:50813)
0927 00:41:55.647384 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:55.647483 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":88,"threads_started":1}
I20260927 00:41:56.029425 15747 raft_consensus.cc:482] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:56.029626 15747 raft_consensus.cc:504] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4b78a88ada461490ce4f69e19f6de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43369 } } peers { permanent_uuid: "84fdfd237d1f4550acc3fa381ce3719e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46695 } } peers { permanent_uuid: "ea25c9d62f834badafac6d614dc9805f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41241 } }
I20260927 00:41:56.030252 15747 leader_election.cc:288] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369), 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695)
W20260927 00:41:56.031304 15312 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b4b78a88ada461490ce4f69e19f6de7 (127.0.98.3:43369): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
W20260927 00:41:56.031422 15314 leader_election.cc:341] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84fdfd237d1f4550acc3fa381ce3719e (127.0.98.2:46695): Not found: Tablet not found: 997b672d089749c39ada7d86e28431c4
I20260927 00:41:56.031487 15314 leader_election.cc:302] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea25c9d62f834badafac6d614dc9805f; no voters: 0b4b78a88ada461490ce4f69e19f6de7, 84fdfd237d1f4550acc3fa381ce3719e
I20260927 00:41:56.031693 15747 raft_consensus.cc:2726] T 997b672d089749c39ada7d86e28431c4 P ea25c9d62f834badafac6d614dc9805f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:41:56.322685 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.322159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.1:46887)
0927 00:41:56.322512 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.322574 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20260927 00:41:56.456660 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.456031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35829 (local address 127.0.98.2:35119)
0927 00:41:56.456436 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.456524 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
I20260927 00:41:56.628419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15262
I20260927 00:41:56.633486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15429
I20260927 00:41:56.638785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15564
2026-09-27T00:41:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19249 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:56 dist-test-slave-xltq krb5kdc[15752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:56 dist-test-slave-xltq krb5kdc[15752](info): set up 2 sockets
Sep 27 00:41:56 dist-test-slave-xltq krb5kdc[15752](info): commencing operation
krb5kdc: starting...
W20260927 00:41:57.628350 392 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:57 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:57.761706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:57.878757 15765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:57.879079 15765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:57.879141 15765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:57.883512 15765 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:41:57.883620 15765 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:41:57.883641 15765 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:41:57.883661 15765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:57.883685 15765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:57.883713 15765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:57.883738 15765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:57.883769 15765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:57.889796 15765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:57.890995 15765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:57.892045 15765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:57.901568 15770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:57.901583 15773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:57.901588 15771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:57.901643 15765 server_base.cc:1034] running on GCE node
I20260927 00:41:57.902673 15765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:57.903900 15765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:57.905118 15765 hybrid_clock.cc:648] HybridClock initialized: now 1790469717905101 us; error 34 us; skew 500 ppm
Sep 27 00:41:57 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:57.908656 15765 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:57.909832 15765 webserver.cc:460] Webserver started at http://127.0.98.62:42659/ using document root <none> and password file <none>
I20260927 00:41:57.910425 15765 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:57.910498 15765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:57.910722 15765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:57.912961 15765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "723488b6b13d46fcb9c3f84fd51d3793"
format_stamp: "Formatted at 2026-09-27 00:41:57 on dist-test-slave-xltq"
server_key: "4025473e62f078cefd4e1c3d5325990c"
server_key_iv: "c341ccd41e86d5ba33915536a1bae5e5"
server_key_version: "encryptionkey@0"
I20260927 00:41:57.913566 15765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "723488b6b13d46fcb9c3f84fd51d3793"
format_stamp: "Formatted at 2026-09-27 00:41:57 on dist-test-slave-xltq"
server_key: "4025473e62f078cefd4e1c3d5325990c"
server_key_iv: "c341ccd41e86d5ba33915536a1bae5e5"
server_key_version: "encryptionkey@0"
I20260927 00:41:57.917784 15765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:41:57.921083 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:57.922683 15765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:57.922869 15765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "723488b6b13d46fcb9c3f84fd51d3793"
format_stamp: "Formatted at 2026-09-27 00:41:57 on dist-test-slave-xltq"
server_key: "4025473e62f078cefd4e1c3d5325990c"
server_key_iv: "c341ccd41e86d5ba33915536a1bae5e5"
server_key_version: "encryptionkey@0"
I20260927 00:41:57.923009 15765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:57.941741 15765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:57.945076 15765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:57.945313 15765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:57.952610 15765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260927 00:41:57.952900 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260927 00:41:57.953615 15765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:57.956574 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.959014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15765
I20260927 00:41:57.959138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:57.959424 392 external_mini_cluster.cc:1428] Setting key 6a0f6d1448da52e4d7643617790fb326
I20260927 00:41:57.963376 15833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap starting.
I20260927 00:41:57.966275 15833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.967352 15833 log.cc:824] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:41:57 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:57.970172 15833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: No bootstrap required, opened a new log
I20260927 00:41:57.974969 15833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:41:57.975287 15833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.975353 15833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.975903 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.961054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.0.98.62:33709)
0927 00:41:57.961492 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:57.961508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:57.961547 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:41:57.962299 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:57.962321 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:57.962696 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:57.963008 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:57.963909 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.965150 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.966423 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.967034 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.967183 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.970230 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:57.970237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:57.970257 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:57.970294 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:57.972425 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:57.972981 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:57.972988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:57.972996 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:57.973163 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:57.973443 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:57.973446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:57.973448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:57.974052 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:57.974268 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.974506 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.974839 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260927 00:41:57.975857 15833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:41:57.976012 15833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:57.976069 15833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:57.976171 15833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.977470 15833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:41:57.977826 15833 leader_election.cc:302] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723488b6b13d46fcb9c3f84fd51d3793; no voters:
I20260927 00:41:57.978160 15833 leader_election.cc:288] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:57.978268 15838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:57.978529 15838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 LEADER]: Becoming Leader. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Running, Role: LEADER
I20260927 00:41:57.978935 15838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:41:57.979418 15833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:57.980778 15840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:41:57.980912 15840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:57.981076 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723488b6b13d46fcb9c3f84fd51d3793. Latest consensus state: current_term: 1 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:41:57.981431 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:57.981530 15852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:57.984670 15852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:57.990921 15852 catalog_manager.cc:1347] Generated new cluster ID: 156d86ec3c0a4ae48860f4f31d61bd57
I20260927 00:41:57.991022 15852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:58.036767 15852 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:58.037869 15852 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:58.045924 15852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Generated new TSK 0
I20260927 00:41:58.046810 15852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:58.091926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:58.210376 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:58.210681 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:58.210726 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:58.215065 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:58.215155 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:58.215217 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:58.220726 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:58.221872 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:58.222960 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:58.233458 15865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.233458 15864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.233527 15867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:58.233688 15859 server_base.cc:1034] running on GCE node
I20260927 00:41:58.234326 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:58.235042 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:58.236263 15859 hybrid_clock.cc:648] HybridClock initialized: now 1790469718236237 us; error 47 us; skew 500 ppm
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:58.239565 15859 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:58.240720 15859 webserver.cc:460] Webserver started at http://127.0.98.1:41073/ using document root <none> and password file <none>
I20260927 00:41:58.241338 15859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:58.241416 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:58.241641 15859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:58.243872 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "b94dfe683d104075bf779e0f06fc6b2b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "4939038ac3506475372504cd5a67d1f8"
server_key_iv: "48335d85bebae854d4055719f05baadb"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.244504 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b94dfe683d104075bf779e0f06fc6b2b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "4939038ac3506475372504cd5a67d1f8"
server_key_iv: "48335d85bebae854d4055719f05baadb"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.248891 15859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:41:58.251883 15874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.253609 15859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:58.253775 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "b94dfe683d104075bf779e0f06fc6b2b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "4939038ac3506475372504cd5a67d1f8"
server_key_iv: "48335d85bebae854d4055719f05baadb"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.253916 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:58.304075 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:58.312944 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:58.313197 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:58.313988 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:58.315178 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:58.315239 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.315299 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:58.315332 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.326792 15859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33491
I20260927 00:41:58.326817 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33491 every 8 connection(s)
I20260927 00:41:58.327800 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:58.330488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15859
I20260927 00:41:58.330592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:58.330814 392 external_mini_cluster.cc:1428] Setting key 631329a0e97a4e5f1d0f2ee7704dfbd2
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:58.343418 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.329706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36609 (local address 127.0.98.62:33709)
0927 00:41:58.329836 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.329840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.330891 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 00:41:58.332694 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.332698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.332755 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.332847 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:58.334395 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.334937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.336152 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.336507 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.336623 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.339675 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:58.339680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:58.339682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:58.339711 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:58.341190 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.341747 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.341751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.341754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.341840 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.342203 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.342206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.342208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.342445 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:58.342546 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.343100 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.343242 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:58.344420 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.330176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:36609)
0927 00:41:58.330693 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.330751 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.332413 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.333168 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.333177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.333603 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:58.334218 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.334237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.335073 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.335076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.336015 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.336027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.336691 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.336696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.336888 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.337654 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:58.337673 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:58.339451 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:58.341326 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.341334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.341355 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.341640 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.341929 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.341932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.341934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.342107 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.342554 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:58.342562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:58.342863 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.343054 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.343427 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260927 00:41:58.345837 15988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:41:58.346124 15988 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:58.346783 15988 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:41:58.348712 15795 ts_manager.cc:194] Registered new tserver with Master: b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:41:58.349884 15795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:58.360597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:58.476078 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:58.476409 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:58.476460 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:58.480674 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:58.480746 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:58.480831 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:58.486636 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:58.487834 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:58.488950 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:58.499305 15998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.499366 16001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.499284 15999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:58.499845 15993 server_base.cc:1034] running on GCE node
I20260927 00:41:58.500373 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:58.501138 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:58.502326 15993 hybrid_clock.cc:648] HybridClock initialized: now 1790469718502299 us; error 48 us; skew 500 ppm
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:58.505803 15993 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:58.506970 15993 webserver.cc:460] Webserver started at http://127.0.98.2:42253/ using document root <none> and password file <none>
I20260927 00:41:58.507581 15993 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:58.507655 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:58.507867 15993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:58.510115 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "eed5f32ec935471b84dc385f7d7c506b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "a73b3d95d686457eb4a181ba810234e5"
server_key_iv: "f8684fa9ca0fb49acca3baa965c48f4e"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.510696 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eed5f32ec935471b84dc385f7d7c506b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "a73b3d95d686457eb4a181ba810234e5"
server_key_iv: "f8684fa9ca0fb49acca3baa965c48f4e"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.514870 15993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:41:58.517889 16008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.519528 15993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:58.519691 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "eed5f32ec935471b84dc385f7d7c506b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "a73b3d95d686457eb4a181ba810234e5"
server_key_iv: "f8684fa9ca0fb49acca3baa965c48f4e"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.519819 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:58.537310 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:58.541283 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:58.541494 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:58.542219 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:58.543112 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:58.543162 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.543217 15993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:58.543243 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.552603 15993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38887
I20260927 00:41:58.552642 16121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38887 every 8 connection(s)
I20260927 00:41:58.553755 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:58.557760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 15993
I20260927 00:41:58.557946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:58.558198 392 external_mini_cluster.cc:1428] Setting key 8d1117bffcac6f549e8bab90ab281ecf
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:58.571592 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.555912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55859 (local address 127.0.98.62:33709)
0927 00:41:58.556064 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.556070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.557174 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 00:41:58.558719 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.558723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.558786 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.558886 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:58.560528 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.561273 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.563043 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.563368 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.563519 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.566980 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:58.566984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:58.566986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:58.567018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:58.569015 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.569729 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.569732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.569734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.569832 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.570278 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.570281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.570282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.570494 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:58.570587 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.571317 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.571454 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:58.572630 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.556403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:55859)
0927 00:41:58.556912 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.556962 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.558393 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.559182 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.559191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.559748 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:58.560373 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.560391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.561432 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.561435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.562882 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.562895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.563603 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.563607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.563831 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.564611 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:58.564634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:58.566670 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:58.569176 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.569183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.569199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.569506 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.569959 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.569962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.569963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.570162 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.570625 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:58.570631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:58.570941 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.571229 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.571551 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":156,"threads_started":1}
I20260927 00:41:58.574726 16122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:41:58.574982 16122 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:58.575505 16122 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:41:58.576756 15795 ts_manager.cc:194] Registered new tserver with Master: eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
I20260927 00:41:58.577394 15795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:58.591274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:58.705538 16127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:58.705850 16127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:58.705896 16127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:58.710124 16127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:58.710208 16127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:58.710270 16127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:58.716636 16127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:41:58.717836 16127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:58.719012 16127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:58.729885 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.729972 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:58.730096 16127 server_base.cc:1034] running on GCE node
W20260927 00:41:58.729880 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:58.730620 16127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:58.731415 16127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:58.732586 16127 hybrid_clock.cc:648] HybridClock initialized: now 1790469718732553 us; error 48 us; skew 500 ppm
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:58.735795 16127 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:58.736901 16127 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260927 00:41:58.737504 16127 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:58.737576 16127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:58.737772 16127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:58.739883 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a755df265544b50a560ae4edbde2209"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "2d331d2815e89a91bd3d24ff0733dc45"
server_key_iv: "ca9533cde4f3c0516ae8d62ad2b4b76c"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.740466 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a755df265544b50a560ae4edbde2209"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "2d331d2815e89a91bd3d24ff0733dc45"
server_key_iv: "ca9533cde4f3c0516ae8d62ad2b4b76c"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.744424 16127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:41:58.747151 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.748620 16127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:58.748741 16127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "8a755df265544b50a560ae4edbde2209"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "2d331d2815e89a91bd3d24ff0733dc45"
server_key_iv: "ca9533cde4f3c0516ae8d62ad2b4b76c"
server_key_version: "encryptionkey@0"
I20260927 00:41:58.748876 16127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:58.786602 16127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:58.790367 16127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:58.790572 16127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:58.791287 16127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:58.792173 16127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:58.792263 16127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.792315 16127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:58.792346 16127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:58.801143 16127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46069
I20260927 00:41:58.801185 16255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46069 every 8 connection(s)
I20260927 00:41:58.802110 16127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:58.808552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16127
I20260927 00:41:58.808671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:58.808969 392 external_mini_cluster.cc:1428] Setting key 071937023fc2b0bb97170ed52d19f66f
Sep 27 00:41:58 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:58.817164 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.804072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60197 (local address 127.0.98.62:33709)
0927 00:41:58.804287 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.804292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.805279 (+ 987us) server_negotiation.cc:407] Connection header received
0927 00:41:58.806628 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.806631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.806717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.806797 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:58.808106 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.808830 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.810356 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.810602 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.810693 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.813549 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:58.813552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:58.813554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:58.813581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:58.815016 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.815543 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.815547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.815550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.815632 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.815963 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.815967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.815968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.816165 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:58.816285 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.816848 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.816996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:41:58.818029 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.804481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:60197)
0927 00:41:58.805120 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.805167 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.806414 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.807050 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.807057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.807442 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:58.807965 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.807979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.808964 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.808967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.810250 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.810263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.810751 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.810755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.810961 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.811645 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:58.811663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:58.813328 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:58.815146 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.815154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.815173 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.815436 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.815716 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:58.815719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:58.815721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:58.815867 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:58.816293 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:58.816300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:58.816613 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.816791 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.817145 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":134,"threads_started":1}
I20260927 00:41:58.819404 16256 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:41:58.819666 16256 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:58.820356 16256 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:41:58.821664 15795 ts_manager.cc:194] Registered new tserver with Master: 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069)
I20260927 00:41:58.822503 15795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60197
I20260927 00:41:58.823097 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:58.833335 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.825659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.0.98.62:33709)
0927 00:41:58.825803 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.825808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.825933 (+ 125us) server_negotiation.cc:407] Connection header received
0927 00:41:58.826053 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.826056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.826099 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.826166 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:58.826888 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.827458 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.828381 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.828613 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.828704 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.830311 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:58.830314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:58.830317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:58.830346 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:58.831687 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.832114 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.832118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.832120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.832219 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:58.832457 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:58.832461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:58.832462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:58.832702 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:58.832825 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.833065 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.833178 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:41:58.836926 15797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:58.839973 15797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:58.855552 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.849473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.0.1:46432)
0927 00:41:58.850745 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.850758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.850846 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.851327 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.851330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.851345 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:58.851428 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.851434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.852543 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.852546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.854108 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.854117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.854981 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.854985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.855103 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855141 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.855252 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.855386 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":55,"thread_start_us":119,"threads_started":1}
I20260927 00:41:58.855571 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.849375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46069 (local address 127.0.0.1:38640)
0927 00:41:58.850306 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.850365 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.850508 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.851111 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.851117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.851140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:58.851295 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.851303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.852806 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.852809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.854105 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.854117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.855228 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.855231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.855320 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855340 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.855417 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.855475 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":52,"thread_start_us":82,"threads_started":1}
I20260927 00:41:58.856086 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.849932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.98.2:38887)
0927 00:41:58.850372 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.850383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.850941 (+ 558us) server_negotiation.cc:407] Connection header received
0927 00:41:58.851025 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.851030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.851209 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.851358 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:58.851525 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.852414 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.854250 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.854862 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.854964 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855134 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.855897 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.855979 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":134,"threads_started":1}
I20260927 00:41:58.856155 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.849581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.0.98.3:46069)
0927 00:41:58.850578 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.850589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.850613 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:41:58.850693 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.850699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.850957 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.851105 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:58.851424 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.852649 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.854257 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.855087 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.855223 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.855471 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.855571 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":150,"threads_started":1}
I20260927 00:41:58.856153 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.850366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.0.98.1:33491)
0927 00:41:58.850770 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.850778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.851078 (+ 300us) server_negotiation.cc:407] Connection header received
0927 00:41:58.851152 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.851156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.851303 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.851412 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:58.851793 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.853005 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.854329 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.855085 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.855191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855333 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.855903 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.856012 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260927 00:41:58.856424 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.850147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.0.1:37674)
0927 00:41:58.850889 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.850901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.850992 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.851541 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.851544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.851562 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:58.851657 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.851662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.853126 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.853129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.854205 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.854218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.855394 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.855397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.855501 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.855524 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.855590 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.855658 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":83,"threads_started":1}
I20260927 00:41:58.859321 16056 tablet_service.cc:1467] Processing CreateTablet for tablet bed7f94454c4448e8750b27a88321760 (DEFAULT_TABLE table=test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:58.859318 16190 tablet_service.cc:1467] Processing CreateTablet for tablet bed7f94454c4448e8750b27a88321760 (DEFAULT_TABLE table=test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:58.859493 15922 tablet_service.cc:1467] Processing CreateTablet for tablet bed7f94454c4448e8750b27a88321760 (DEFAULT_TABLE table=test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:58.860502 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed7f94454c4448e8750b27a88321760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:58.860510 16190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed7f94454c4448e8750b27a88321760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:58.860502 16056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed7f94454c4448e8750b27a88321760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:58.867102 16273 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap starting.
I20260927 00:41:58.868746 16274 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Bootstrap starting.
I20260927 00:41:58.869854 16273 tablet_bootstrap.cc:654] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:58.870695 16273 log.cc:824] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:58.871340 16274 tablet_bootstrap.cc:654] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:58.871426 16275 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Bootstrap starting.
I20260927 00:41:58.872283 16274 log.cc:824] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:58.873116 16273 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: No bootstrap required, opened a new log
I20260927 00:41:58.873353 16273 ts_tablet_manager.cc:1397] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:41:58.873615 16275 tablet_bootstrap.cc:654] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:58.874591 16274 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: No bootstrap required, opened a new log
I20260927 00:41:58.874814 16274 ts_tablet_manager.cc:1397] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:58.874837 16275 log.cc:824] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:58.876940 16275 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: No bootstrap required, opened a new log
I20260927 00:41:58.877137 16275 ts_tablet_manager.cc:1397] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:41:58.878202 16273 raft_consensus.cc:348] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.878541 16273 raft_consensus.cc:374] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:58.878611 16273 raft_consensus.cc:729] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94dfe683d104075bf779e0f06fc6b2b, State: Initialized, Role: FOLLOWER
I20260927 00:41:58.878811 16274 raft_consensus.cc:348] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.879097 16274 raft_consensus.cc:374] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:58.879153 16274 raft_consensus.cc:729] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed5f32ec935471b84dc385f7d7c506b, State: Initialized, Role: FOLLOWER
I20260927 00:41:58.879105 16273 consensus_queue.cc:260] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.879662 16274 consensus_queue.cc:260] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.880460 16122 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260927 00:41:58.880765 16274 ts_tablet_manager.cc:1428] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:41:58.880548 16275 raft_consensus.cc:348] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.880859 16275 raft_consensus.cc:374] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:58.880923 16275 raft_consensus.cc:729] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a755df265544b50a560ae4edbde2209, State: Initialized, Role: FOLLOWER
I20260927 00:41:58.881349 16275 consensus_queue.cc:260] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.880311 16273 ts_tablet_manager.cc:1428] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:41:58.881405 15988 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260927 00:41:58.882262 16256 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260927 00:41:58.882405 16275 ts_tablet_manager.cc:1428] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:41:58.953639 16281 raft_consensus.cc:482] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:58.953827 16281 raft_consensus.cc:504] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.954932 16281 leader_election.cc:288] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887), b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:41:58.959511 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.955679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56841 (local address 127.0.98.1:33491)
0927 00:41:58.955788 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.955792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.955922 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:41:58.956045 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.956048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.956103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.956178 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:58.956448 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.956951 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.958287 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.958950 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.959051 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.959123 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.959296 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.959377 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:41:58.959527 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.955454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.98.3:56841)
0927 00:41:58.955817 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.955831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.955944 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.956220 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.956223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.956239 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:58.956354 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.956359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.957069 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.957072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.958160 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.958171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.959057 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.959061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.959168 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.959195 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.959272 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.959350 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":37,"thread_start_us":117,"threads_started":1}
I20260927 00:41:58.959681 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.955202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.98.3:42959)
0927 00:41:58.955369 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:58.955387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:58.955497 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:58.955864 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:58.955866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:58.955891 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:58.956013 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.956019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.957083 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.957086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.958356 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:58.958366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.959287 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.959291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:58.959414 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.959436 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:58.959503 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:41:58.959575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":65}
I20260927 00:41:58.960120 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.955302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42959 (local address 127.0.98.2:38887)
0927 00:41:58.955438 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.955443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.955459 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:58.955630 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.955633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.955682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.955747 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:58.956127 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.956942 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.958495 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.959088 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.959178 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.959231 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.959906 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.959999 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:41:58.960191 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed7f94454c4448e8750b27a88321760" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94dfe683d104075bf779e0f06fc6b2b" is_pre_election: true
I20260927 00:41:58.960675 15942 raft_consensus.cc:2454] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 0.
I20260927 00:41:58.960677 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed7f94454c4448e8750b27a88321760" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed5f32ec935471b84dc385f7d7c506b" is_pre_election: true
I20260927 00:41:58.960980 16076 raft_consensus.cc:2454] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 0.
I20260927 00:41:58.961190 16143 leader_election.cc:302] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a755df265544b50a560ae4edbde2209, b94dfe683d104075bf779e0f06fc6b2b; no voters:
I20260927 00:41:58.961510 16281 raft_consensus.cc:2781] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:58.961591 16281 raft_consensus.cc:482] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:58.961638 16281 raft_consensus.cc:3037] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:58.962733 16281 raft_consensus.cc:504] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.963102 16281 leader_election.cc:288] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 election: Requested vote from peers eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887), b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:41:58.963497 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed7f94454c4448e8750b27a88321760" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed5f32ec935471b84dc385f7d7c506b"
I20260927 00:41:58.963641 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed7f94454c4448e8750b27a88321760" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94dfe683d104075bf779e0f06fc6b2b"
I20260927 00:41:58.963651 16076 raft_consensus.cc:3037] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:58.963775 15942 raft_consensus.cc:3037] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:58.965121 15942 raft_consensus.cc:2454] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 1.
I20260927 00:41:58.965121 16076 raft_consensus.cc:2454] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 1.
I20260927 00:41:58.965538 16143 leader_election.cc:302] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b; no voters:
I20260927 00:41:58.965770 16281 raft_consensus.cc:2781] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:58.966041 16281 raft_consensus.cc:686] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 1 LEADER]: Becoming Leader. State: Replica: 8a755df265544b50a560ae4edbde2209, State: Running, Role: LEADER
I20260927 00:41:58.966588 16281 consensus_queue.cc:237] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:41:58.970518 15795 catalog_manager.cc:5495] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a755df265544b50a560ae4edbde2209 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8a755df265544b50a560ae4edbde2209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:58.985550 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:58.982021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.98.3:46069)
0927 00:41:58.982154 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:58.982158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:58.982173 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:58.982251 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:58.982254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:58.982316 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:58.982405 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:58.982639 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.983133 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.984014 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:58.984263 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:58.984362 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:58.984688 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:58.985144 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:58.985254 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:58.985326 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:41:58.985387 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:58.991864 16076 raft_consensus.cc:1261] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Refusing update from remote peer 8a755df265544b50a560ae4edbde2209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:58.991930 15942 raft_consensus.cc:1261] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Refusing update from remote peer 8a755df265544b50a560ae4edbde2209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:58.992794 16281 consensus_queue.cc:1035] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:58.993042 16283 consensus_queue.cc:1035] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:59.002856 16286 mvcc.cc:205] Tried to move back new op lower bound from 7333763968982675456 to 7333763968890474496. Current Snapshot: MvccSnapshot[applied={T|T < 7333763968982675456}]
I20260927 00:41:59.003150 16288 mvcc.cc:205] Tried to move back new op lower bound from 7333763968982675456 to 7333763968890474496. Current Snapshot: MvccSnapshot[applied={T|T < 7333763968982675456}]
I20260927 00:41:59.011893 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:59.012085 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:59.014644 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Sending DeleteTablet for 3 replicas of tablet bed7f94454c4448e8750b27a88321760
I20260927 00:41:59.015555 16190 tablet_service.cc:1514] Processing DeleteTablet for tablet bed7f94454c4448e8750b27a88321760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38640
I20260927 00:41:59.015641 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet bed7f94454c4448e8750b27a88321760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37674
I20260927 00:41:59.015678 16056 tablet_service.cc:1514] Processing DeleteTablet for tablet bed7f94454c4448e8750b27a88321760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46432
I20260927 00:41:59.016287 16296 tablet_replica.cc:330] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: stopping tablet replica
I20260927 00:41:59.016388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15859
I20260927 00:41:59.016593 16297 tablet_replica.cc:330] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: stopping tablet replica
I20260927 00:41:59.017028 16297 raft_consensus.cc:2229] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:59.017241 16296 raft_consensus.cc:2229] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:59.017617 16297 raft_consensus.cc:2258] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:59.017627 16296 raft_consensus.cc:2258] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:59.019170 16297 ts_tablet_manager.cc:1905] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:59.021412 16296 ts_tablet_manager.cc:1905] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:59.021844 16297 ts_tablet_manager.cc:1918] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:59.021953 16297 log.cc:1196] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/bed7f94454c4448e8750b27a88321760
I20260927 00:41:59.022260 16297 ts_tablet_manager.cc:1939] T bed7f94454c4448e8750b27a88321760 P eed5f32ec935471b84dc385f7d7c506b: Deleting consensus metadata
I20260927 00:41:59.023838 15781 catalog_manager.cc:4841] TS eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): tablet bed7f94454c4448e8750b27a88321760 (table test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]) successfully deleted
W20260927 00:41:59.024446 15781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33491 (error 108)
I20260927 00:41:59.024971 16296 ts_tablet_manager.cc:1918] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:59.025091 16296 log.cc:1196] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/bed7f94454c4448e8750b27a88321760
I20260927 00:41:59.025260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15993
I20260927 00:41:59.025432 16296 ts_tablet_manager.cc:1939] T bed7f94454c4448e8750b27a88321760 P 8a755df265544b50a560ae4edbde2209: Deleting consensus metadata
I20260927 00:41:59.026918 15783 catalog_manager.cc:4841] TS 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): tablet bed7f94454c4448e8750b27a88321760 (table test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]) successfully deleted
I20260927 00:41:59.029109 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:59.028750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.0.1:37688)
0927 00:41:59.028961 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:59.029031 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:41:59.029301 15781 catalog_manager.cc:4584] TS b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bed7f94454c4448e8750b27a88321760: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33491: connect: Connection refused (error 111)
I20260927 00:41:59.032871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16127
I20260927 00:41:59.038661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 15765
I20260927 00:42:19.044701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=42659
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:19.187305 16300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.187712 16300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.187786 16300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.193030 16300 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:42:19.193145 16300 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:42:19.193171 16300 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:42:19.193192 16300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:19.193218 16300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.193245 16300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.193277 16300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:19.193318 16300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:19.199980 16300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:19.201651 16300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.203305 16300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.221669 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.221747 16300 server_base.cc:1034] running on GCE node
W20260927 00:42:19.221681 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.221669 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.223174 16300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.225241 16300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.226613 16300 hybrid_clock.cc:648] HybridClock initialized: now 1790469739226576 us; error 58 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.230679 16300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:42:19.232144 16300 webserver.cc:460] Webserver started at http://127.0.98.62:42659/ using document root <none> and password file <none>
I20260927 00:42:19.233019 16300 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.233135 16300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.238840 16300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 00:42:19.242499 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.244799 16300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:42:19.245075 16300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "723488b6b13d46fcb9c3f84fd51d3793"
format_stamp: "Formatted at 2026-09-27 00:41:57 on dist-test-slave-xltq"
server_key: "4025473e62f078cefd4e1c3d5325990c"
server_key_iv: "c341ccd41e86d5ba33915536a1bae5e5"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.245962 16300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.278811 16300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.282614 16300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.282821 16300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.291726 16300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260927 00:42:19.292256 16368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260927 00:42:19.293393 16300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:19.293730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16300
I20260927 00:42:19.294296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33491
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=41073
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:42:19.300547 16369 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:42:19.303442 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap starting.
I20260927 00:42:19.327124 16369 log.cc:824] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:19.338919 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:42:19.339397 16369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap complete.
I20260927 00:42:19.345083 16369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:19.345649 16369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.346257 16369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:19.346421 16369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:19.346509 16369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:19.346616 16369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:42:19.349397 16369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:19.349805 16369 leader_election.cc:302] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723488b6b13d46fcb9c3f84fd51d3793; no voters:
I20260927 00:42:19.350230 16369 leader_election.cc:288] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:42:19.350376 16373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:42:19.350636 16373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 LEADER]: Becoming Leader. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Running, Role: LEADER
I20260927 00:42:19.351011 16373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:19.351666 16369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:19.352648 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:42:19.352770 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:19.353391 16375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723488b6b13d46fcb9c3f84fd51d3793. Latest consensus state: current_term: 2 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:42:19.353483 16375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:19.353950 16382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:19.358994 16382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:19.359715 16382 catalog_manager.cc:1259] Loaded cluster ID: 156d86ec3c0a4ae48860f4f31d61bd57
I20260927 00:42:19.359776 16382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:19.362457 16382 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:19.363206 16382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Loaded TSK: 0
I20260927 00:42:19.376323 16382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Generated new TSK 1
I20260927 00:42:19.376631 16382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:42:19.438521 16371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.438802 16371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.438862 16371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.443121 16371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.443192 16371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.443280 16371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:19.448812 16371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41073
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:19.449980 16371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.451234 16371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.462409 16396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.462527 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.462416 16397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.462670 16371 server_base.cc:1034] running on GCE node
I20260927 00:42:19.463205 16371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.463898 16371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.465080 16371 hybrid_clock.cc:648] HybridClock initialized: now 1790469739465054 us; error 41 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.467882 16371 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:42:19.468973 16371 webserver.cc:460] Webserver started at http://127.0.98.1:41073/ using document root <none> and password file <none>
I20260927 00:42:19.469580 16371 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.469657 16371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.473892 16371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:42:19.476265 16406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.477663 16371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:19.477793 16371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "b94dfe683d104075bf779e0f06fc6b2b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "4939038ac3506475372504cd5a67d1f8"
server_key_iv: "48335d85bebae854d4055719f05baadb"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.478381 16371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.492858 16371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.496299 16371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.496495 16371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.497192 16371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:19.498435 16413 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:42:19.502715 16371 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:42:19.502792 16371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 00:42:19.502851 16371 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:42:19.504803 16371 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:42:19.504879 16371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:19.505177 16413 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap starting.
I20260927 00:42:19.515761 16371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33491
I20260927 00:42:19.515802 16520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33491 every 8 connection(s)
I20260927 00:42:19.517249 16371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:19.522673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16371
I20260927 00:42:19.523180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38887
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=42253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:42:19.526543 16413 log.cc:824] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:19.535215 16413 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:42:19.535645 16413 tablet_bootstrap.cc:492] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap complete.
I20260927 00:42:19.536870 16413 ts_tablet_manager.cc:1397] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260927 00:42:19.538066 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.519780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:34119)
0927 00:42:19.520460 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.520541 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.521624 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.523628 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.523638 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.524173 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.524986 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.525011 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.526454 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.526458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.527623 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.527640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.528481 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.528485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.528682 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.529357 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.529377 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.531287 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.534083 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.534090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.534105 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.534515 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.535009 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.535012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.535014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.535185 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.536097 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.536104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.536441 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.536648 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.537102 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":171,"threads_started":1}
I20260927 00:42:19.538067 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.519733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34119 (local address 127.0.98.62:33709)
0927 00:42:19.520105 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.520125 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.520798 (+ 673us) server_negotiation.cc:407] Connection header received
0927 00:42:19.522142 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.522156 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.522590 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.523038 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.525214 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.526261 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.527773 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.528334 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.528507 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.531571 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.531580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.531601 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.531644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.533899 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.534689 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.534696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.534702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.534880 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.535362 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.535366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.535368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.535951 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.536125 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.536831 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.537194 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":112,"threads_started":1}
I20260927 00:42:19.539914 16413 raft_consensus.cc:348] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.540408 16413 raft_consensus.cc:729] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94dfe683d104075bf779e0f06fc6b2b, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.541057 16413 consensus_queue.cc:260] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.541393 16521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:42:19.541661 16521 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:19.542186 16413 ts_tablet_manager.cc:1428] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:42:19.542230 16521 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:42:19.545207 16327 ts_manager.cc:194] Registered new tserver with Master: b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:42:19.549460 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34119
I20260927 00:42:19.552481 16521 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:42:19.560518 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.548932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.0.1:50984)
0927 00:42:19.549876 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.549913 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.550076 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.550713 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.550719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.551169 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.551311 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.551320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.552753 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.552757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.553693 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.553706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.554434 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.554453 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.554571 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.555124 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.555136 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.556766 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.558703 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.558714 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.558720 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.558950 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.559245 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.559248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.559250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.559329 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.559792 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.559798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.559932 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.560060 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.560319 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":115,"threads_started":1}
I20260927 00:42:19.560518 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.549051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.98.1:33491)
0927 00:42:19.549509 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.549521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.550032 (+ 511us) server_negotiation.cc:407] Connection header received
0927 00:42:19.550314 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.550319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.550542 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.550795 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.551442 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.552626 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.553831 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.554335 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.554432 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.556941 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.556953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.556963 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.556988 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.558580 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.559063 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.559069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.559071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.559158 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.559411 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.559414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.559416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.559679 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.559798 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.560165 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.560356 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":125,"threads_started":1}
I20260927 00:42:19.562012 16455 tablet_service.cc:1514] Processing DeleteTablet for tablet bed7f94454c4448e8750b27a88321760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50984
I20260927 00:42:19.563028 16530 tablet_replica.cc:330] stopping tablet replica
I20260927 00:42:19.563328 16530 raft_consensus.cc:2229] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:19.563576 16530 raft_consensus.cc:2258] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:19.564653 16530 ts_tablet_manager.cc:1905] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:19.568440 16530 ts_tablet_manager.cc:1918] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:19.568555 16530 log.cc:1196] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal/wals/bed7f94454c4448e8750b27a88321760
I20260927 00:42:19.568778 16530 ts_tablet_manager.cc:1939] T bed7f94454c4448e8750b27a88321760 P b94dfe683d104075bf779e0f06fc6b2b: Deleting consensus metadata
I20260927 00:42:19.571300 16317 catalog_manager.cc:4841] TS b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491): tablet bed7f94454c4448e8750b27a88321760 (table test-table [id=ce2810af3aaf4c84a7847e1dfd41eb3e]) successfully deleted
W20260927 00:42:19.648895 16525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.649189 16525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.649233 16525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.653702 16525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.653782 16525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.653846 16525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:19.659487 16525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42253
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:19.660614 16525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.661617 16525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.672106 16538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.672317 16536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.672113 16535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.672171 16525 server_base.cc:1034] running on GCE node
I20260927 00:42:19.672956 16525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.673647 16525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.674834 16525 hybrid_clock.cc:648] HybridClock initialized: now 1790469739674813 us; error 36 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.677919 16525 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:42:19.678934 16525 webserver.cc:460] Webserver started at http://127.0.98.2:42253/ using document root <none> and password file <none>
I20260927 00:42:19.679543 16525 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.679631 16525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.684571 16525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:19.687120 16545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.688874 16525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:19.689038 16525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "eed5f32ec935471b84dc385f7d7c506b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "a73b3d95d686457eb4a181ba810234e5"
server_key_iv: "f8684fa9ca0fb49acca3baa965c48f4e"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.689677 16525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.703864 16525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.707557 16525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.707774 16525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.708431 16525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:19.709311 16525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:19.709367 16525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.709421 16525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:19.709446 16525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.718784 16525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38887
I20260927 00:42:19.718842 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38887 every 8 connection(s)
I20260927 00:42:19.719940 16525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:19.723914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16525
I20260927 00:42:19.724462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46069
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=32781
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:19.736268 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.722127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60961 (local address 127.0.98.62:33709)
0927 00:42:19.722285 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.722290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.723365 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 00:42:19.725022 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.725026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.725094 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.725208 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.727117 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.727732 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.729091 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.729342 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.729449 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.732395 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.732399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.732401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.732433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.733976 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.734635 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.734638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.734640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.734716 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.735046 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.735049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.735050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.735240 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.735325 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.735895 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.736051 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:42:19.737236 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.722655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:60961)
0927 00:42:19.723188 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.723238 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.724755 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.725940 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.725949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.726354 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.726896 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.726912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.727923 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.727926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.728946 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.728958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.729544 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.729549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.729746 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.730394 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.730412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.732168 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.734183 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.734190 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.734209 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.734521 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.734798 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.734801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.734803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.734948 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.735334 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.735341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.735633 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.735821 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.736180 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260927 00:42:19.738427 16659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:42:19.738668 16659 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:19.739172 16659 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:42:19.740319 16327 ts_manager.cc:194] Registered new tserver with Master: eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
I20260927 00:42:19.740896 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60961
W20260927 00:42:19.840655 16662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.840941 16662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.840986 16662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.845185 16662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.845268 16662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.845329 16662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:19.850883 16662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32781
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:19.852063 16662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.853193 16662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.863309 16668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.863343 16667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.863399 16662 server_base.cc:1034] running on GCE node
W20260927 00:42:19.863334 16670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.864079 16662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.864837 16662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.866026 16662 hybrid_clock.cc:648] HybridClock initialized: now 1790469739865992 us; error 54 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.869076 16662 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:42:19.870091 16662 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260927 00:42:19.870643 16662 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.870718 16662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.874890 16662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:42:19.877276 16677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.878728 16662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:42:19.878856 16662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "8a755df265544b50a560ae4edbde2209"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "2d331d2815e89a91bd3d24ff0733dc45"
server_key_iv: "ca9533cde4f3c0516ae8d62ad2b4b76c"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.879488 16662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.895290 16662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.899055 16662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.899227 16662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.899876 16662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:19.900761 16662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:19.900810 16662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.900851 16662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:19.900870 16662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.909880 16662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46069
I20260927 00:42:19.909909 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46069 every 8 connection(s)
I20260927 00:42:19.910876 16662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:19.911520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16662
I20260927 00:42:19.926047 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.912393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.98.62:33709)
0927 00:42:19.912561 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.912566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.917852 (+ 5286us) server_negotiation.cc:407] Connection header received
0927 00:42:19.917950 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.917954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.918030 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.918157 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.919213 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.919984 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.920925 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.921253 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.921373 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.922896 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.922900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.922902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.922928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.924394 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.924789 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.924792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.924794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.924874 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.925125 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.925128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.925130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.925338 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.925542 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.925788 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.925897 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":2432}
Sep 27 00:42:19 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:19.932034 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.918479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43657 (local address 127.0.98.62:33709)
0927 00:42:19.918799 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.918804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.919834 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 00:42:19.921398 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.921401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.921461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.921573 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.923099 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.923884 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.925272 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.925584 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.925668 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.928698 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.928701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.928703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.928730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.930059 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.930608 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.930610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.930611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.930721 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.931009 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.931011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.931012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.931200 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.931271 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.931784 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.931895 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"spinlock_wait_cycles":4736,"thread_start_us":112,"threads_started":1}
I20260927 00:42:19.932881 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.919027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:43657)
0927 00:42:19.919652 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.919718 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.921128 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.921771 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.921780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.922309 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.922942 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.922970 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.924078 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.924082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.925160 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.925172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.925760 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.925764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.926093 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.926747 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.926768 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.928460 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.930224 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.930231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.930245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.930523 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.930793 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.930795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.930796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.930936 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.931282 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.931288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.931551 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.931710 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.931982 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":150,"threads_started":1}
I20260927 00:42:19.934262 16791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260927 00:42:19.934502 16791 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:19.935021 16791 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260927 00:42:19.936340 16327 ts_manager.cc:194] Registered new tserver with Master: 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069)
I20260927 00:42:19.936924 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43657
I20260927 00:42:19.940261 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:42:19.944408 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.940865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.98.62:33709)
0927 00:42:19.941075 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.941080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.941201 (+ 121us) server_negotiation.cc:407] Connection header received
0927 00:42:19.941374 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.941376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.941512 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.941603 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:42:19.941846 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.942365 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.943223 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.943453 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.943543 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.943841 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:42:19.943981 (+ 140us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:42:19.944274 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45304: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:42:19.944480 16794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45304: token expired
W20260927 00:42:19.944962 16259 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:33709)
I20260927 00:42:19.952571 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.945367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.0.98.62:33709)
0927 00:42:19.945465 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.945468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.945479 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:42:19.945598 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.945600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.945638 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.945697 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.946261 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.946781 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.947661 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.947867 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.947944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.949278 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.949281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.949283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.949317 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.950870 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.951285 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.951288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.951289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.951373 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.951719 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.951722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.951724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.951919 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.952104 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.952305 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.952413 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:42:19.955302 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:42:19.958617 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:42:19.967559 16455 tablet_service.cc:1467] Processing CreateTablet for tablet a382301638be48c086e20e37de5cf595 (DEFAULT_TABLE table=test-table [id=1514e17bf5134504bedc907652ee8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:19.968263 16455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382301638be48c086e20e37de5cf595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:19.971006 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.965881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46069 (local address 127.0.0.1:60280)
0927 00:42:19.966013 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.966029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.966118 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.967141 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.967144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.967163 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:19.967274 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.967280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.968619 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.968623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.969740 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.969750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.970547 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.970551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.970646 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.970686 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.970755 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.970860 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:42:19.971019 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.966110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.0.98.3:46069)
0927 00:42:19.966513 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.966522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.966544 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:42:19.966617 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.966624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.966887 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.967152 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:19.967364 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.968503 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.969872 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.970430 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.970537 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.970691 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.970805 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.970894 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260927 00:42:19.971047 16413 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap starting.
I20260927 00:42:19.971927 16413 tablet_bootstrap.cc:654] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:19.972692 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.966560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.0.1:33962)
0927 00:42:19.966778 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.966791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.966875 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.967643 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.967646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.967663 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:19.967754 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.967760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.969035 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.969038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.970416 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.970427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.972269 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.972272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.972377 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.972401 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.972468 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.972545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260927 00:42:19.972954 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.966769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.0.98.2:38887)
0927 00:42:19.967235 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.967248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.967268 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:42:19.967337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.967342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.967521 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.967650 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:19.967958 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.968899 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.971182 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.971916 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.972031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.972181 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.972526 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.972627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":91,"threads_started":1}
I20260927 00:42:19.973554 16413 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: No bootstrap required, opened a new log
I20260927 00:42:19.973651 16413 ts_tablet_manager.cc:1397] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:19.974128 16413 raft_consensus.cc:348] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.974257 16413 raft_consensus.cc:374] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:19.974294 16413 raft_consensus.cc:729] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94dfe683d104075bf779e0f06fc6b2b, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.974426 16413 consensus_queue.cc:260] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.974686 16413 ts_tablet_manager.cc:1428] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:42:19.974861 16725 tablet_service.cc:1467] Processing CreateTablet for tablet a382301638be48c086e20e37de5cf595 (DEFAULT_TABLE table=test-table [id=1514e17bf5134504bedc907652ee8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:19.975921 16725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382301638be48c086e20e37de5cf595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:19.979512 16593 tablet_service.cc:1467] Processing CreateTablet for tablet a382301638be48c086e20e37de5cf595 (DEFAULT_TABLE table=test-table [id=1514e17bf5134504bedc907652ee8545]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:19.980575 16593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a382301638be48c086e20e37de5cf595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:19.983603 16804 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Bootstrap starting.
I20260927 00:42:19.985800 16804 tablet_bootstrap.cc:654] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:19.986831 16804 log.cc:824] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:19.988334 16806 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Bootstrap starting.
I20260927 00:42:19.988957 16804 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: No bootstrap required, opened a new log
I20260927 00:42:19.989185 16804 ts_tablet_manager.cc:1397] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 00:42:19.991168 16806 tablet_bootstrap.cc:654] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:19.992261 16806 log.cc:824] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:19.993350 16804 raft_consensus.cc:348] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.993676 16804 raft_consensus.cc:374] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:19.993738 16804 raft_consensus.cc:729] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a755df265544b50a560ae4edbde2209, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.994244 16804 consensus_queue.cc:260] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.994807 16806 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: No bootstrap required, opened a new log
I20260927 00:42:19.995020 16806 ts_tablet_manager.cc:1397] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:42:19.995003 16791 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260927 00:42:19.995412 16804 ts_tablet_manager.cc:1428] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 00:42:19.998203 16806 raft_consensus.cc:348] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.998550 16806 raft_consensus.cc:374] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:19.998625 16806 raft_consensus.cc:729] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed5f32ec935471b84dc385f7d7c506b, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.999114 16806 consensus_queue.cc:260] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:19.999919 16659 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260927 00:42:20.000140 16806 ts_tablet_manager.cc:1428] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 00:42:20.018680 16522 tablet.cc:2369] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:42:20.107805 16808 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:20.108018 16808 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:20.109261 16808 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887), b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:42:20.113658 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.109583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.98.3:42441)
0927 00:42:20.109783 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.109810 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.109923 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.110270 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.110273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.110295 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.110422 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.110429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.111443 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.111446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.112380 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.112390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.113204 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.113208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.113321 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.113348 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.113427 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.113500 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 00:42:20.113667 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.109757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42441 (local address 127.0.98.2:38887)
0927 00:42:20.109924 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.109928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.109944 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:42:20.110002 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.110004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.110146 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.110238 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.110588 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.111313 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.112513 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.113075 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.113164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.113223 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.113478 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.113549 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:42:20.114121 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.109823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.98.3:42357)
0927 00:42:20.110192 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.110204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.110280 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.110600 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.110603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.110628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.110764 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.110771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.111724 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.111727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.112762 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.112773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.113709 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.113713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.113818 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.113848 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.113921 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.113989 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260927 00:42:20.114267 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.109974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42357 (local address 127.0.98.1:33491)
0927 00:42:20.110226 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.110231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.110247 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:42:20.110377 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.110381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.110445 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.110555 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.110915 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.111577 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.112901 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.113558 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.113656 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.113843 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.114040 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.114139 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":63,"threads_started":1}
I20260927 00:42:20.114478 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382301638be48c086e20e37de5cf595" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed5f32ec935471b84dc385f7d7c506b" is_pre_election: true
I20260927 00:42:20.114840 16613 raft_consensus.cc:2454] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 0.
I20260927 00:42:20.114933 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382301638be48c086e20e37de5cf595" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94dfe683d104075bf779e0f06fc6b2b" is_pre_election: true
I20260927 00:42:20.115234 16475 raft_consensus.cc:2454] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 0.
I20260927 00:42:20.115346 16678 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b; no voters:
I20260927 00:42:20.115656 16808 raft_consensus.cc:2781] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:42:20.115751 16808 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:42:20.115782 16808 raft_consensus.cc:3037] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.116883 16808 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:20.117218 16808 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 election: Requested vote from peers eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887), b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491)
I20260927 00:42:20.117666 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382301638be48c086e20e37de5cf595" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed5f32ec935471b84dc385f7d7c506b"
I20260927 00:42:20.117666 16475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a382301638be48c086e20e37de5cf595" candidate_uuid: "8a755df265544b50a560ae4edbde2209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94dfe683d104075bf779e0f06fc6b2b"
I20260927 00:42:20.117802 16613 raft_consensus.cc:3037] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.117807 16475 raft_consensus.cc:3037] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.119102 16613 raft_consensus.cc:2454] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 1.
I20260927 00:42:20.119102 16475 raft_consensus.cc:2454] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a755df265544b50a560ae4edbde2209 in term 1.
I20260927 00:42:20.119417 16678 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b; no voters:
I20260927 00:42:20.119593 16808 raft_consensus.cc:2781] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:20.119789 16808 raft_consensus.cc:686] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 1 LEADER]: Becoming Leader. State: Replica: 8a755df265544b50a560ae4edbde2209, State: Running, Role: LEADER
I20260927 00:42:20.120110 16808 consensus_queue.cc:237] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:20.123625 16327 catalog_manager.cc:5495] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a755df265544b50a560ae4edbde2209 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8a755df265544b50a560ae4edbde2209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } health_report { overall_health: UNKNOWN } } }
I20260927 00:42:20.136981 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.133365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.98.3:46069)
0927 00:42:20.133486 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.133489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.133502 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:20.133649 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.133653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.133711 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.133798 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:42:20.134083 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.134624 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.135462 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.135684 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.135775 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.136173 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:42:20.136638 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:42:20.136725 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.136814 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.136868 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:42:20.143201 16475 raft_consensus.cc:1261] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Refusing update from remote peer 8a755df265544b50a560ae4edbde2209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:20.143220 16613 raft_consensus.cc:1261] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Refusing update from remote peer 8a755df265544b50a560ae4edbde2209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:20.143970 16812 consensus_queue.cc:1035] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:20.144179 16808 consensus_queue.cc:1035] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:20.149525 16814 mvcc.cc:205] Tried to move back new op lower bound from 7333764055618404352 to 7333764055534706688. Current Snapshot: MvccSnapshot[applied={T|T < 7333764055618404352}]
I20260927 00:42:20.151404 16815 mvcc.cc:205] Tried to move back new op lower bound from 7333764055618404352 to 7333764055534706688. Current Snapshot: MvccSnapshot[applied={T|T < 7333764055618404352 or (T in {7333764055618404352})}]
I20260927 00:42:20.160135 16327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:20.160347 16327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:20.163101 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Sending DeleteTablet for 3 replicas of tablet a382301638be48c086e20e37de5cf595
I20260927 00:42:20.164016 16455 tablet_service.cc:1514] Processing DeleteTablet for tablet a382301638be48c086e20e37de5cf595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50984
I20260927 00:42:20.164077 16725 tablet_service.cc:1514] Processing DeleteTablet for tablet a382301638be48c086e20e37de5cf595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60280
I20260927 00:42:20.164086 16593 tablet_service.cc:1514] Processing DeleteTablet for tablet a382301638be48c086e20e37de5cf595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33962
I20260927 00:42:20.164310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16371
I20260927 00:42:20.164976 16825 tablet_replica.cc:330] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: stopping tablet replica
I20260927 00:42:20.165271 16825 raft_consensus.cc:2229] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:42:20.165624 16825 raft_consensus.cc:2258] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:20.166801 16826 tablet_replica.cc:330] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: stopping tablet replica
I20260927 00:42:20.167171 16826 raft_consensus.cc:2229] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:20.167220 16825 ts_tablet_manager.cc:1905] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:20.167537 16826 raft_consensus.cc:2258] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:20.169289 16826 ts_tablet_manager.cc:1905] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:20.170156 16825 ts_tablet_manager.cc:1918] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:20.170259 16825 log.cc:1196] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/a382301638be48c086e20e37de5cf595
I20260927 00:42:20.170593 16825 ts_tablet_manager.cc:1939] T a382301638be48c086e20e37de5cf595 P 8a755df265544b50a560ae4edbde2209: Deleting consensus metadata
I20260927 00:42:20.171944 16319 catalog_manager.cc:4841] TS 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): tablet a382301638be48c086e20e37de5cf595 (table test-table [id=1514e17bf5134504bedc907652ee8545]) successfully deleted
I20260927 00:42:20.172278 16826 ts_tablet_manager.cc:1918] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:20.172386 16826 log.cc:1196] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/a382301638be48c086e20e37de5cf595
I20260927 00:42:20.172683 16826 ts_tablet_manager.cc:1939] T a382301638be48c086e20e37de5cf595 P eed5f32ec935471b84dc385f7d7c506b: Deleting consensus metadata
W20260927 00:42:20.172957 16317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33491 (error 108)
I20260927 00:42:20.173076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16525
I20260927 00:42:20.175350 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.174833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33491 (local address 127.0.0.1:51000)
0927 00:42:20.175212 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.175276 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:42:20.175490 16317 catalog_manager.cc:4584] TS b94dfe683d104075bf779e0f06fc6b2b (127.0.98.1:33491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a382301638be48c086e20e37de5cf595: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33491: connect: Connection refused (error 111)
I20260927 00:42:20.179738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16662
I20260927 00:42:20.180979 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.180661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.0.1:33978)
0927 00:42:20.180805 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.180875 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:42:20.187198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16300
I20260927 00:42:40.193703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=42659
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:40.312493 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:40.312793 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:40.312844 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:40.317056 16828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:42:40.317133 16828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:42:40.317149 16828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:42:40.317162 16828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:40.317179 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:40.317198 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:40.317212 16828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:40.317227 16828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:40.322764 16828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:40.323952 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:40.325063 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:40.334692 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:40.334710 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:40.334692 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:40.334934 16828 server_base.cc:1034] running on GCE node
I20260927 00:42:40.335855 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:40.337172 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:40.338413 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790469760338387 us; error 50 us; skew 500 ppm
Sep 27 00:42:40 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:40.341904 16828 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:42:40.343078 16828 webserver.cc:460] Webserver started at http://127.0.98.62:42659/ using document root <none> and password file <none>
I20260927 00:42:40.343703 16828 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:40.343775 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:40.348377 16828 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:42:40.351259 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:40.353178 16828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:42:40.353360 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "723488b6b13d46fcb9c3f84fd51d3793"
format_stamp: "Formatted at 2026-09-27 00:41:57 on dist-test-slave-xltq"
server_key: "4025473e62f078cefd4e1c3d5325990c"
server_key_iv: "c341ccd41e86d5ba33915536a1bae5e5"
server_key_version: "encryptionkey@0"
I20260927 00:42:40.353974 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:40.373469 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:40.377302 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:40.377513 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:40.385694 16828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260927 00:42:40.386116 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260927 00:42:40.386929 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:40.392891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16828
I20260927 00:42:40.393224 16896 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:42:40.393417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33491
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=41073
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:42:40.395752 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap starting.
I20260927 00:42:40.418709 16896 log.cc:824] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:40.435051 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:42:40.435593 16896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Bootstrap complete.
I20260927 00:42:40.440682 16896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:40.441237 16896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Initialized, Role: FOLLOWER
I20260927 00:42:40.441869 16896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:40.442023 16896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:40.442102 16896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:40.442212 16896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:42:40.444973 16896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:40.445438 16896 leader_election.cc:302] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723488b6b13d46fcb9c3f84fd51d3793; no voters:
I20260927 00:42:40.445829 16896 leader_election.cc:288] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:42:40.445963 16900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:42:40.446285 16900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [term 3 LEADER]: Becoming Leader. State: Replica: 723488b6b13d46fcb9c3f84fd51d3793, State: Running, Role: LEADER
I20260927 00:42:40.446785 16900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260927 00:42:40.447487 16896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:40.448616 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:42:40.448837 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:40.449270 16902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723488b6b13d46fcb9c3f84fd51d3793. Latest consensus state: current_term: 3 leader_uuid: "723488b6b13d46fcb9c3f84fd51d3793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723488b6b13d46fcb9c3f84fd51d3793" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260927 00:42:40.449380 16902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:40.449360 16909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:40.456261 16909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:40.457260 16909 catalog_manager.cc:1259] Loaded cluster ID: 156d86ec3c0a4ae48860f4f31d61bd57
I20260927 00:42:40.457413 16909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:40.460696 16909 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:40.461954 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Loaded TSK: 0
I20260927 00:42:40.462155 16909 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 723488b6b13d46fcb9c3f84fd51d3793: Loaded TSK: 1
F20260927 00:42:40.463248 16909 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad95e199ad google::LogMessage::Fail() at ??:0
@ 0x7fad95e1c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fad95e194de google::LogMessage::Flush() at ??:0
@ 0x7fad95e1aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad98add18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fad98ad5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fada07827bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fada075c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fada075d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fada079edf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fada07bd82b std::function<>::operator()() at ??:0
@ 0x7fada075cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fada075df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fada075a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fada079e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fad9806b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad9806bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fad9806d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fad9805c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad996a06db start_thread at ??:0
@ 0x7fad9491761f clone at ??:0
W20260927 00:42:40.537328 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:40.537714 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:40.537786 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:40.544643 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:40.544729 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:40.544821 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:40.553674 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41073
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:40.555197 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:40.556710 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:40.569585 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:40.569729 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:40.569585 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:40.569986 16898 server_base.cc:1034] running on GCE node
I20260927 00:42:40.570677 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:40.571591 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:40.572814 16898 hybrid_clock.cc:648] HybridClock initialized: now 1790469760572778 us; error 60 us; skew 500 ppm
Sep 27 00:42:40 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:40.576987 16898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:42:40.578441 16898 webserver.cc:460] Webserver started at http://127.0.98.1:41073/ using document root <none> and password file <none>
I20260927 00:42:40.579275 16898 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:40.579367 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:40.585835 16898 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:42:40.589787 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:40.591895 16898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 00:42:40.592127 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "b94dfe683d104075bf779e0f06fc6b2b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "4939038ac3506475372504cd5a67d1f8"
server_key_iv: "48335d85bebae854d4055719f05baadb"
server_key_version: "encryptionkey@0"
I20260927 00:42:40.593034 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:40.626590 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:40.631873 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:40.632129 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:40.632961 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:40.634677 16960 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:42:40.641216 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:42:40.641310 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 00:42:40.641399 16898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:42:40.643368 16898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:42:40.643443 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:42:40.643726 16960 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap starting.
I20260927 00:42:40.655337 16898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33491
I20260927 00:42:40.655843 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33491 every 8 connection(s)
I20260927 00:42:40.656939 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:40.661823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 16898
I20260927 00:42:40.662453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38887
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=42253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:42:40.666803 16960 log.cc:824] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:40.677794 16960 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:42:40.678303 16960 tablet_bootstrap.cc:492] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Bootstrap complete.
I20260927 00:42:40.679714 16960 ts_tablet_manager.cc:1397] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.015s
I20260927 00:42:40.683319 16960 raft_consensus.cc:348] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:40.683985 16960 raft_consensus.cc:729] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94dfe683d104075bf779e0f06fc6b2b, State: Initialized, Role: FOLLOWER
I20260927 00:42:40.684639 16960 consensus_queue.cc:260] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:40.685961 16960 ts_tablet_manager.cc:1428] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 00:42:40.732990 17070 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:40.660626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:33269)
0927 00:42:40.662262 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.662343 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:40.663804 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:40.731531 (+ 67727us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":233,"threads_started":1}
W20260927 00:42:40.733875 17068 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:42:40.735369 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:40.734844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:38553)
0927 00:42:40.735121 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.735250 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:42:40.736591 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:40.736269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:45227)
0927 00:42:40.736422 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.736501 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 00:42:40.736837 17068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:42:40.790978 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:40.791281 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:40.791342 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:40.795598 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:40.795682 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:40.795769 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:40.801486 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42253
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:40.802640 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:40.803669 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:40.813838 17078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:40.813896 17071 server_base.cc:1034] running on GCE node
W20260927 00:42:40.813813 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:40.813804 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:40.814499 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:40.815132 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:40.816269 17071 hybrid_clock.cc:648] HybridClock initialized: now 1790469760816241 us; error 40 us; skew 500 ppm
Sep 27 00:42:40 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:40.819293 17071 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:42:40.820451 17071 webserver.cc:460] Webserver started at http://127.0.98.2:42253/ using document root <none> and password file <none>
I20260927 00:42:40.821064 17071 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:40.821138 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:40.825652 17071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:42:40.828508 17088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:40.830165 17071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:42:40.830338 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "eed5f32ec935471b84dc385f7d7c506b"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "a73b3d95d686457eb4a181ba810234e5"
server_key_iv: "f8684fa9ca0fb49acca3baa965c48f4e"
server_key_version: "encryptionkey@0"
I20260927 00:42:40.830969 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:40.841765 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:40.845091 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:40.845287 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:40.845947 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:40.846830 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:40.846881 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:40.846935 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:40.846964 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:40.855865 17071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38887
I20260927 00:42:40.855921 17201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38887 every 8 connection(s)
I20260927 00:42:40.856899 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:40.861382 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:40.859415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:43841)
0927 00:42:40.859926 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.860044 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
W20260927 00:42:40.861840 17202 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
I20260927 00:42:40.862468 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:40.862224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56861)
0927 00:42:40.862367 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.862414 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:42:40.862985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17071
I20260927 00:42:40.863222 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:40.862987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:52035)
0927 00:42:40.863128 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:40.863172 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 00:42:40.863386 17202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:42:40.863452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46069
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33709
--webserver_port=32781
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:40.979504 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:40.979821 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:40.979887 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:40.984483 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:40.984575 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:40.984702 17205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:40.990751 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32781
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:40.991964 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:40.993090 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:42:41.003458 17205 server_base.cc:1034] running on GCE node
W20260927 00:42:41.003407 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:41.003379 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:41.003379 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:41.004119 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:41.004873 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:41.006079 17205 hybrid_clock.cc:648] HybridClock initialized: now 1790469761006048 us; error 48 us; skew 500 ppm
Sep 27 00:42:41 dist-test-slave-xltq krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:41.009368 17205 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:42:41.010479 17205 webserver.cc:460] Webserver started at http://127.0.98.3:32781/ using document root <none> and password file <none>
I20260927 00:42:41.011072 17205 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:41.011139 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:41.015484 17205 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:41.018129 17220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:41.019572 17205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:42:41.019718 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "8a755df265544b50a560ae4edbde2209"
format_stamp: "Formatted at 2026-09-27 00:41:58 on dist-test-slave-xltq"
server_key: "2d331d2815e89a91bd3d24ff0733dc45"
server_key_iv: "ca9533cde4f3c0516ae8d62ad2b4b76c"
server_key_version: "encryptionkey@0"
I20260927 00:42:41.020365 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:41.038546 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:41.042523 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:41.042754 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:41.043504 17205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:41.044530 17205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:41.044589 17205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:41.044639 17205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:41.044673 17205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:41.054566 17205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46069
I20260927 00:42:41.054620 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46069 every 8 connection(s)
I20260927 00:42:41.055668 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:41.059957 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:41.058022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:51953)
0927 00:42:41.058546 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:41.058678 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
W20260927 00:42:41.060674 17334 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33709 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33709: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
I20260927 00:42:41.060806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17205
I20260927 00:42:41.061621 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:41.061325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:45679)
0927 00:42:41.061487 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:41.061541 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 00:42:41.062368 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:41.062160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:56171)
0927 00:42:41.062274 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:41.062316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:42:41.062561 17334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:42:41.738142 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:41.737570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:58193)
0927 00:42:41.737908 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:41.737988 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260927 00:42:41.864573 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:41.864072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56717)
0927 00:42:41.864377 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:41.864442 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260927 00:42:42.063488 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.063088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:42047)
0927 00:42:42.063325 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:42.063380 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260927 00:42:42.273530 17341 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:42.273729 17341 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:42.275079 17341 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069), eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
Sep 27 00:42:42 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:42:42 dist-test-slave-xltq krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:42:42.290427 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.275384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38887 (local address 127.0.98.1:37937)
0927 00:42:42.275684 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:42.275699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:42.275800 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:42.278398 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:42.278401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:42.278663 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:42.279238 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:42.279254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.280708 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.280711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:42.281778 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:42.281790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.282633 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.282637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:42.282868 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:42.283497 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:42.283515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:42.285606 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:42.287797 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:42.287804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:42.287821 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:42.288133 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:42.288526 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:42.288531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:42.288672 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:42.289351 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:42.289358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:42.289646 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 00:42:42.289826 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:42:42.290228 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":145,"spinlock_wait_cycles":2816,"thread_start_us":71,"threads_started":1}
I20260927 00:42:42.290442 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.275602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37937 (local address 127.0.98.2:38887)
0927 00:42:42.275947 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:42.275972 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:42.276018 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:42:42.277002 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:42.277012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:42.277505 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:42.277870 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:42.279452 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.280535 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.281976 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.282484 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.282667 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:42.285803 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:42.285808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:42.285820 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:42.285855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:42.287653 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288284 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:42.288290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:42.288294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:42.288432 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288761 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:42.288764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:42.288766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:42.289222 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:42.289342 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:42.289962 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 00:42:42.290227 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260927 00:42:42.290442 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.275489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53365 (local address 127.0.98.3:46069)
0927 00:42:42.276015 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:42.276033 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:42.276063 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:42:42.277002 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:42.277012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:42.277505 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:42.277870 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:42.279466 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.280535 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.281933 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.282457 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.282638 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:42.285633 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:42.285639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:42.285653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:42.285685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:42.287754 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288338 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:42.288343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:42.288348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:42.288483 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288762 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:42.288765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:42.288766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:42.289221 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:42.289342 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:42.289965 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 00:42:42.290228 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":80,"threads_started":1}
I20260927 00:42:42.290472 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.275320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46069 (local address 127.0.98.1:53365)
0927 00:42:42.275641 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:42.275657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:42.275778 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:42.278103 (+ 2325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:42.278108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:42.278639 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:42.279239 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:42.279256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.280705 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.280708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:42.281777 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:42.281790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:42.282596 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:42.282600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:42.282862 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:42.283540 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:42.283552 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:42.285376 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:42.287971 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:42.287974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:42.287975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:42.288188 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:42.288574 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:42.288578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:42.288579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:42.288675 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:42.289351 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:42.289358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:42.289647 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 00:42:42.289827 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:42:42.290312 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":484,"spinlock_wait_cycles":40448,"thread_start_us":83,"threads_started":1}
W20260927 00:42:42.293546 16956 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
W20260927 00:42:42.293752 16954 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
I20260927 00:42:42.293828 16954 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b94dfe683d104075bf779e0f06fc6b2b; no voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b
I20260927 00:42:42.294067 17341 raft_consensus.cc:2726] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:42:42.739483 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.739091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:53941)
0927 00:42:42.739272 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:42.739393 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 00:42:42.865825 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:42.865396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:51611)
0927 00:42:42.865667 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:42.865729 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260927 00:42:43.064873 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:43.064424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:46359)
0927 00:42:43.064686 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:43.064750 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260927 00:42:43.740804 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:43.740365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:32957)
0927 00:42:43.740635 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:43.740700 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260927 00:42:43.867208 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:43.866742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56695)
0927 00:42:43.867051 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:43.867111 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":73,"threads_started":1}
I20260927 00:42:44.066169 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:44.065682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:60687)
0927 00:42:44.066010 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:44.066072 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260927 00:42:44.156381 17351 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:44.156554 17351 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:44.156955 17351 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069), eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
W20260927 00:42:44.157856 16954 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
W20260927 00:42:44.158032 16956 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
I20260927 00:42:44.158110 16956 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b94dfe683d104075bf779e0f06fc6b2b; no voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b
I20260927 00:42:44.158327 17351 raft_consensus.cc:2726] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:42:44.742101 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:44.741661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:55227)
0927 00:42:44.741946 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:44.742004 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260927 00:42:44.869122 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:44.868469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:34669)
0927 00:42:44.868872 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:44.868985 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":84,"threads_started":1}
I20260927 00:42:45.067693 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:45.067070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:34659)
0927 00:42:45.067469 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:45.067537 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":99,"threads_started":1}
I20260927 00:42:45.743974 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:45.743315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:46619)
0927 00:42:45.743716 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:45.743840 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
I20260927 00:42:45.870891 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:45.870141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:57431)
0927 00:42:45.870609 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:45.870734 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":109,"threads_started":1}
I20260927 00:42:46.069145 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:46.068661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:59665)
0927 00:42:46.068971 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:46.069042 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260927 00:42:46.618672 17358 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:46.618850 17358 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:46.619417 17358 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069), eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
W20260927 00:42:46.620236 16956 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
W20260927 00:42:46.620329 16954 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
I20260927 00:42:46.620421 16954 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b94dfe683d104075bf779e0f06fc6b2b; no voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b
I20260927 00:42:46.620568 17358 raft_consensus.cc:2726] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:42:46.745512 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:46.744982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:54133)
0927 00:42:46.745286 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:46.745349 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260927 00:42:46.872483 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:46.871925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:41915)
0927 00:42:46.872267 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:46.872341 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260927 00:42:47.070539 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:47.070041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:55459)
0927 00:42:47.070340 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:47.070432 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260927 00:42:47.746774 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:47.746306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:53623)
0927 00:42:47.746614 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:47.746680 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260927 00:42:47.873864 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:47.873403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:41051)
0927 00:42:47.873708 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:47.873768 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":70,"threads_started":1}
I20260927 00:42:48.072576 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:48.071817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:42905)
0927 00:42:48.072341 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:48.072409 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":117,"threads_started":1}
I20260927 00:42:48.748526 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:48.747815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:39131)
0927 00:42:48.748306 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:48.748398 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":112,"threads_started":1}
I20260927 00:42:48.875663 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:48.874964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:41033)
0927 00:42:48.875443 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:48.875542 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":110,"threads_started":1}
I20260927 00:42:49.074348 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:49.073693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:51535)
0927 00:42:49.074096 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:49.074193 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260927 00:42:49.750243 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:49.749631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:47293)
0927 00:42:49.750019 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:49.750112 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":105,"threads_started":1}
I20260927 00:42:49.877483 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:49.876802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:45493)
0927 00:42:49.877291 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:49.877353 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":132,"threads_started":1}
I20260927 00:42:50.076185 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:50.075532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:55227)
0927 00:42:50.075925 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:50.076074 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260927 00:42:50.752043 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:50.751474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:52603)
0927 00:42:50.751815 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:50.751903 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260927 00:42:50.879283 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:50.878664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56599)
0927 00:42:50.879049 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:50.879149 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260927 00:42:51.077822 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:51.077276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:45715)
0927 00:42:51.077637 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:51.077721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260927 00:42:51.271859 17374 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:51.272135 17374 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:51.272746 17374 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069), eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
W20260927 00:42:51.273874 16956 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
W20260927 00:42:51.274127 16954 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
I20260927 00:42:51.274299 16954 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b94dfe683d104075bf779e0f06fc6b2b; no voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b
I20260927 00:42:51.274503 17374 raft_consensus.cc:2726] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:42:51.753829 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:51.753146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:46733)
0927 00:42:51.753592 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:51.753693 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":87,"threads_started":1}
I20260927 00:42:51.881038 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:51.880455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:41491)
0927 00:42:51.880807 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:51.880908 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":83,"threads_started":1}
I20260927 00:42:52.079588 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:52.078984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:39449)
0927 00:42:52.079354 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:52.079450 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
I20260927 00:42:52.755594 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:52.754964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:41403)
0927 00:42:52.755383 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:52.755468 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":156,"threads_started":1}
I20260927 00:42:52.882967 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:52.882332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:47179)
0927 00:42:52.882727 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:52.882825 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":90,"threads_started":1}
I20260927 00:42:53.081315 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:53.080755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:36047)
0927 00:42:53.081089 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:53.081180 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":88,"threads_started":1}
I20260927 00:42:53.560889 17381 raft_consensus.cc:482] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:53.561120 17381 raft_consensus.cc:504] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a755df265544b50a560ae4edbde2209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46069 } } peers { permanent_uuid: "eed5f32ec935471b84dc385f7d7c506b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38887 } } peers { permanent_uuid: "b94dfe683d104075bf779e0f06fc6b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33491 } }
I20260927 00:42:53.561661 17381 leader_election.cc:288] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069), eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887)
W20260927 00:42:53.562708 16956 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a755df265544b50a560ae4edbde2209 (127.0.98.3:46069): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
W20260927 00:42:53.562907 16954 leader_election.cc:341] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eed5f32ec935471b84dc385f7d7c506b (127.0.98.2:38887): Not found: Tablet not found: a382301638be48c086e20e37de5cf595
I20260927 00:42:53.562973 16954 leader_election.cc:302] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b94dfe683d104075bf779e0f06fc6b2b; no voters: 8a755df265544b50a560ae4edbde2209, eed5f32ec935471b84dc385f7d7c506b
I20260927 00:42:53.563174 17381 raft_consensus.cc:2726] T a382301638be48c086e20e37de5cf595 P b94dfe683d104075bf779e0f06fc6b2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:42:53.757138 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:53.756599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:43769)
0927 00:42:53.756961 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:53.757025 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260927 00:42:53.884727 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:53.884059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:56047)
0927 00:42:53.884505 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:53.884593 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":185,"threads_started":1}
I20260927 00:42:54.083155 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:54.082576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:60175)
0927 00:42:54.082940 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:54.083028 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260927 00:42:54.758939 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:54.758335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:40117)
0927 00:42:54.758723 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:54.758830 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260927 00:42:54.886479 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:54.885879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:40811)
0927 00:42:54.886281 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:54.886371 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260927 00:42:55.084916 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:55.084298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:36825)
0927 00:42:55.084720 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:55.084807 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":130,"threads_started":1}
I20260927 00:42:55.761129 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:55.760511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:56507)
0927 00:42:55.760912 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:55.761002 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":149,"threads_started":1}
I20260927 00:42:55.888090 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:55.887627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:42939)
0927 00:42:55.887937 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:55.887993 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260927 00:42:56.068266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 16898
I20260927 00:42:56.075671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17071
I20260927 00:42:56.082154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17205
2026-09-27T00:42:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59442 ms)
[----------] 30 tests from SecurityITest (247077 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:42:56 dist-test-slave-xltq krb5kdc[17392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:42:56 dist-test-slave-xltq krb5kdc[17392](info): set up 2 sockets
Sep 27 00:42:56 dist-test-slave-xltq krb5kdc[17392](info): commencing operation
krb5kdc: starting...
W20260927 00:42:57.064121 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:42:57.188045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:57.302058 17405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:57.302385 17405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:57.302443 17405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:57.306664 17405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:57.306743 17405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:57.306772 17405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:57.306794 17405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:57.306814 17405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:57.312436 17405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:57.313575 17405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:57.314713 17405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:42:57.323910 17405 server_base.cc:1034] running on GCE node
W20260927 00:42:57.323859 17411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:57.323925 17413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:57.323992 17410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:57.324741 17405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:57.326021 17405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:57.327208 17405 hybrid_clock.cc:648] HybridClock initialized: now 1790469777327189 us; error 35 us; skew 500 ppm
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:57.330205 17405 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:42:57.331229 17405 webserver.cc:460] Webserver started at http://127.0.98.62:40221/ using document root <none> and password file <none>
I20260927 00:42:57.331809 17405 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:57.331879 17405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:57.332082 17405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:57.334354 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "a65ab18d7de8400bbc42ec3462cf58cb"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "754b8267dc79432cf0bd9f0495617aee"
server_key_iv: "e009d871ba6192fc8ff3e5c5e01052ae"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.334941 17405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "a65ab18d7de8400bbc42ec3462cf58cb"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "754b8267dc79432cf0bd9f0495617aee"
server_key_iv: "e009d871ba6192fc8ff3e5c5e01052ae"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.339138 17405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:42:57.342010 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.343429 17405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:57.343573 17405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "a65ab18d7de8400bbc42ec3462cf58cb"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "754b8267dc79432cf0bd9f0495617aee"
server_key_iv: "e009d871ba6192fc8ff3e5c5e01052ae"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.343708 17405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:57.357270 17405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:57.360632 17405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:57.360833 17405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:57.367754 17405 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41281
I20260927 00:42:57.367774 17472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41281 every 8 connection(s)
I20260927 00:42:57.368806 17405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:57.371759 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:57.374804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17405
I20260927 00:42:57.374926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:42:57.375207 392 external_mini_cluster.cc:1428] Setting key 5f61a84df6536906da97b52ebf4b50c4
I20260927 00:42:57.377794 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: Bootstrap starting.
I20260927 00:42:57.380398 17473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:57.381213 17473 log.cc:824] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:57.383563 17473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: No bootstrap required, opened a new log
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:57.387037 17473 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260927 00:42:57.387326 17473 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:57.387390 17473 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65ab18d7de8400bbc42ec3462cf58cb, State: Initialized, Role: FOLLOWER
I20260927 00:42:57.387964 17473 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260927 00:42:57.388136 17473 raft_consensus.cc:388] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:57.388247 17473 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:57.388355 17473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:57.389343 17473 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260927 00:42:57.389731 17473 leader_election.cc:302] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a65ab18d7de8400bbc42ec3462cf58cb; no voters:
I20260927 00:42:57.390053 17473 leader_election.cc:288] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:42:57.390177 17478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:57.390378 17478 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [term 1 LEADER]: Becoming Leader. State: Replica: a65ab18d7de8400bbc42ec3462cf58cb, State: Running, Role: LEADER
I20260927 00:42:57.390650 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:57.376832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.98.62:41281)
0927 00:42:57.377321 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:57.377338 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:57.377377 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:42:57.378081 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:57.378101 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:57.378473 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:57.378787 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:57.379743 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.380890 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.381961 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.382442 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.382594 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:57.385051 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:57.385056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:57.385071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:57.385098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:57.387369 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.387938 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.387944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.387948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.388090 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.388456 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.388459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.388461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.388871 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:57.389069 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:57.389341 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 00:42:57.389648 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":134,"threads_started":1}
I20260927 00:42:57.390758 17478 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } }
I20260927 00:42:57.391340 17473 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:57.391990 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65ab18d7de8400bbc42ec3462cf58cb. Latest consensus state: current_term: 1 leader_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260927 00:42:57.392098 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:57.393115 17484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:57.393682 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65ab18d7de8400bbc42ec3462cf58cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41281 } } }
I20260927 00:42:57.393798 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:57.396420 17484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:57.402763 17484 catalog_manager.cc:1347] Generated new cluster ID: 83ff35deee9e40439959c4484c140c62
I20260927 00:42:57.402841 17484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:57.412525 17484 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:42:57.413508 17484 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:57.421962 17484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: Generated new TSK 0
I20260927 00:42:57.422734 17484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:42:57.459285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:57.576261 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:57.576570 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:57.576617 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:57.580886 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:57.580966 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:57.581030 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:57.586745 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:57.587905 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:57.589006 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:42:57.599344 17499 server_base.cc:1034] running on GCE node
W20260927 00:42:57.599191 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:57.599200 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:57.599181 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:57.599993 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:57.600746 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:57.601939 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790469777601920 us; error 40 us; skew 500 ppm
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:57.604959 17499 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:42:57.606007 17499 webserver.cc:460] Webserver started at http://127.0.98.1:35809/ using document root <none> and password file <none>
I20260927 00:42:57.606568 17499 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:57.606645 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:57.606865 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:57.609056 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "5358fa8870004f6d8a12dcfe5f07f931"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "f240d5f6d494e1a196edc90a9af36176"
server_key_iv: "f181327b7c6af84a5a87b45713cc6afb"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.609615 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5358fa8870004f6d8a12dcfe5f07f931"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "f240d5f6d494e1a196edc90a9af36176"
server_key_iv: "f181327b7c6af84a5a87b45713cc6afb"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.613476 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:42:57.616057 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.617440 17499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:57.617563 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "5358fa8870004f6d8a12dcfe5f07f931"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "f240d5f6d494e1a196edc90a9af36176"
server_key_iv: "f181327b7c6af84a5a87b45713cc6afb"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.617676 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:57.658676 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:57.662357 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:57.662568 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:57.663239 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:57.664134 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:57.664182 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.664263 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:57.664291 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.673290 17499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35843
I20260927 00:42:57.673357 17627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35843 every 8 connection(s)
I20260927 00:42:57.674242 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:57.676724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17499
I20260927 00:42:57.676855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:57.677145 392 external_mini_cluster.cc:1428] Setting key d86affdcfebecb8bbcc7e320b0d94b5c
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:42:57.690227 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:57.676254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41371 (local address 127.0.98.62:41281)
0927 00:42:57.676491 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:57.676497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:57.677737 (+ 1240us) server_negotiation.cc:407] Connection header received
0927 00:42:57.678943 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:57.678948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:57.679003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:57.679081 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:57.680355 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.680912 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.682187 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.682450 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.682572 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:57.685535 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:57.685540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:57.685543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:57.685577 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:57.687311 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.688029 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.688033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.688034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.688133 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.688717 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.688721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.688723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.688970 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:57.689056 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:57.689952 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 00:42:57.690084 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 00:42:57.691246 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:57.676683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.1:41371)
0927 00:42:57.677385 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:57.677446 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:57.678714 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:57.679238 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:57.679246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:57.679629 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:57.680164 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:57.680179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.681064 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.681068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:57.682030 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:57.682042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.682639 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.682644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:57.682833 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:57.683495 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:57.683515 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:57.685302 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:57.687477 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:57.687483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:57.687499 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:57.687903 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:57.688384 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:57.688387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:57.688389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:57.688593 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:57.689089 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:57.689097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:57.689610 (+ 513us) client_negotiation.cc:769] Sending connection context
0927 00:42:57.689857 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 00:42:57.690273 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":194,"threads_started":1}
I20260927 00:42:57.692755 17628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260927 00:42:57.693045 17628 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:57.693776 17628 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260927 00:42:57.695622 17437 ts_manager.cc:194] Registered new tserver with Master: 5358fa8870004f6d8a12dcfe5f07f931 (127.0.98.1:35843)
I20260927 00:42:57.696990 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:42:57.708359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:57.826169 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:57.826471 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:57.826516 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:57.831157 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:57.831243 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:57.831306 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:57.837268 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:57.838429 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:57.839488 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:57.849848 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:57.849979 17633 server_base.cc:1034] running on GCE node
W20260927 00:42:57.849841 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:57.850176 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:57.850694 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:57.851390 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:57.852548 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790469777852526 us; error 35 us; skew 500 ppm
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:57.855700 17633 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:42:57.856961 17633 webserver.cc:460] Webserver started at http://127.0.98.2:45693/ using document root <none> and password file <none>
I20260927 00:42:57.857539 17633 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:57.857609 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:57.857815 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:57.859916 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "a260215deb6942179efe6fa4b16fc636"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "7d065e3de8ea8d68c7e5e2e2a872d2d2"
server_key_iv: "4cda1bf034468cc1f9fdb55f4d87216a"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.860515 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a260215deb6942179efe6fa4b16fc636"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "7d065e3de8ea8d68c7e5e2e2a872d2d2"
server_key_iv: "4cda1bf034468cc1f9fdb55f4d87216a"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.864764 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:42:57.867560 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.869182 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:57.869333 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "a260215deb6942179efe6fa4b16fc636"
format_stamp: "Formatted at 2026-09-27 00:42:57 on dist-test-slave-xltq"
server_key: "7d065e3de8ea8d68c7e5e2e2a872d2d2"
server_key_iv: "4cda1bf034468cc1f9fdb55f4d87216a"
server_key_version: "encryptionkey@0"
I20260927 00:42:57.869459 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:57.881130 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:57.884812 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:57.885016 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:57.885705 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:57.886597 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:57.886649 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.886705 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:57.886734 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:57.896468 17633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35655
I20260927 00:42:57.896574 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35655 every 8 connection(s)
I20260927 00:42:57.897517 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:57.905162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17633
I20260927 00:42:57.905288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:57.905578 392 external_mini_cluster.cc:1428] Setting key 572c7417c2c0a742edcfc8c88258f8f8
Sep 27 00:42:57 dist-test-slave-xltq krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:57.913126 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:57.899473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54165 (local address 127.0.98.62:41281)
0927 00:42:57.899620 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:57.899624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:57.900566 (+ 942us) server_negotiation.cc:407] Connection header received
0927 00:42:57.901946 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:57.901949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:57.901999 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:57.902080 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:57.903303 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.903886 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.905508 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.905821 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.905926 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:57.909163 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:57.909167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:57.909169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:57.909198 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:57.910824 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.911444 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.911447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.911449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.911533 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:57.911899 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:57.911902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:57.911903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:57.912104 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:57.912191 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:57.912807 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:42:57.912947 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:42:57.913993 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:57.899839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.2:54165)
0927 00:42:57.900411 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:57.900460 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:57.901708 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:57.902245 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:57.902252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:57.902634 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:57.903132 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:57.903146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.904036 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.904039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:57.905338 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:57.905353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:57.906016 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:57.906020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:57.906205 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:57.906989 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:57.907015 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:57.908945 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:57.910990 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:57.910997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:57.911017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:57.911323 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:57.911620 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:57.911623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:57.911625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:57.911796 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:57.912239 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:57.912247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:57.912531 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 00:42:57.912706 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 00:42:57.913089 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:42:57.915318 17762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260927 00:42:57.915571 17762 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:57.916185 17762 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260927 00:42:57.917469 17437 ts_manager.cc:194] Registered new tserver with Master: a260215deb6942179efe6fa4b16fc636 (127.0.98.2:35655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:42:57.918181 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:42:57.934911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:58.048961 17767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:58.049253 17767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:58.049316 17767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:58.053499 17767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:58.053577 17767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:58.053665 17767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:58.059247 17767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:58.060436 17767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:58.061462 17767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:58.071421 17775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.071507 17773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.071473 17772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:58.071519 17767 server_base.cc:1034] running on GCE node
I20260927 00:42:58.072126 17767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:58.072973 17767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:58.074167 17767 hybrid_clock.cc:648] HybridClock initialized: now 1790469778074155 us; error 31 us; skew 500 ppm
Sep 27 00:42:58 dist-test-slave-xltq krb5kdc[17392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:58.077153 17767 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:42:58.078189 17767 webserver.cc:460] Webserver started at http://127.0.98.3:37253/ using document root <none> and password file <none>
I20260927 00:42:58.078776 17767 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:58.078846 17767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:58.079077 17767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:58.081319 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "618971d7b2854781bffd0fe584fa929f"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "6d17650e0ff46f735644168d5ba3bb49"
server_key_iv: "04d8dcd04683451fee7fc05db37f797e"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.081924 17767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "618971d7b2854781bffd0fe584fa929f"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "6d17650e0ff46f735644168d5ba3bb49"
server_key_iv: "04d8dcd04683451fee7fc05db37f797e"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.086036 17767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:42:58.088922 17782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.090456 17767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:58.090612 17767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "618971d7b2854781bffd0fe584fa929f"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "6d17650e0ff46f735644168d5ba3bb49"
server_key_iv: "04d8dcd04683451fee7fc05db37f797e"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.090744 17767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:58.111052 17767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:58.114547 17767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:58.114740 17767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:58.115386 17767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:58.116358 17767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:58.116410 17767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.116451 17767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:58.116465 17767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.125460 17767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42169
I20260927 00:42:58.125483 17895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42169 every 8 connection(s)
I20260927 00:42:58.126384 17767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:58.131937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17767
I20260927 00:42:58.132052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:58.132406 392 external_mini_cluster.cc:1428] Setting key 473d4f2425de45597c6e3ca771899163
Sep 27 00:42:58 dist-test-slave-xltq krb5kdc[17392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:58.141167 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40021 (local address 127.0.98.62:41281)
0927 00:42:58.128550 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.128555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.129591 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 00:42:58.130867 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.130871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.130921 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.130998 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:58.132308 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.133010 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.134541 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.134813 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.134898 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.137426 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:58.137429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:58.137431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:58.137459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:58.138915 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:58.139402 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:58.139405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:58.139406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:58.139486 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:58.139886 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:58.139889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:58.139890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:58.140090 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:58.140192 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.140914 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.141034 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:42:58.142136 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.128877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41281 (local address 127.0.98.3:40021)
0927 00:42:58.129413 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.129463 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.130629 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.131156 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.131163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.131603 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:58.132132 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.132151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.133153 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.133157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.134385 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.134398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.134954 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.134958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.135115 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.135684 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:58.135700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:58.137233 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:58.139027 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:58.139032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:58.139048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:58.139307 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:58.139589 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:58.139592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:58.139594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:58.139753 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:58.140345 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:58.140351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:58.140637 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.140821 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.141128 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260927 00:42:58.143435 17896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41281
I20260927 00:42:58.143680 17896 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:58.144256 17896 heartbeater.cc:511] Master 127.0.98.62:41281 requested a full tablet report, sending...
I20260927 00:42:58.145406 17437 ts_manager.cc:194] Registered new tserver with Master: 618971d7b2854781bffd0fe584fa929f (127.0.98.3:42169)
I20260927 00:42:58.146080 17437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40021
I20260927 00:42:58.146665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:42:58.157100 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.148848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.0.98.62:41281)
0927 00:42:58.149019 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.149023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.149177 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:42:58.149225 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.149227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.149277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.149428 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:58.150212 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.150788 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.151632 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.151869 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.151949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.153429 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:58.153432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:58.153434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:58.153461 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:58.155130 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:58.155611 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:58.155614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:58.155616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:58.155706 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:58.156025 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:58.156028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:58.156030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:58.156438 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:58.156564 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.156818 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.156929 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 00:42:58.160748 17434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:42:58.163775 17434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:42:58.179759 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.175023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.0.98.1:35843)
0927 00:42:58.175545 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.175556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.175574 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:42:58.175629 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.175633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.175778 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.176111 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:58.176281 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.177297 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.178369 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179025 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179125 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179290 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.179481 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.179591 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":71,"threads_started":1}
I20260927 00:42:58.179738 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.173336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35655 (local address 127.0.0.1:42350)
0927 00:42:58.173730 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.173790 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.173939 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.174810 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.174816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.174841 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:58.175087 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.175097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.176674 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.176677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.178155 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.178166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179371 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.179465 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179487 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.179543 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.179594 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
I20260927 00:42:58.179738 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.174794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35843 (local address 127.0.0.1:43860)
0927 00:42:58.175252 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.175265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.175358 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.175914 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.175917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.175930 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:58.175993 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.175998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.177437 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.177440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.178249 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.178261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179144 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.179269 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179309 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.179419 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.179557 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":25,"thread_start_us":107,"threads_started":1}
I20260927 00:42:58.180413 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.173569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.0.98.2:35655)
0927 00:42:58.173999 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.174017 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.174040 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:42:58.174175 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.174180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.174403 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.174565 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:58.175339 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.176522 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.178301 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179112 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179221 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179381 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.180126 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.180265 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":130,"threads_started":1}
I20260927 00:42:58.180558 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.174550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42169 (local address 127.0.0.1:37410)
0927 00:42:58.175151 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.175165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.175246 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.175798 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.175801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.175827 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:58.175937 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.175943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.177329 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.177333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.178472 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.178481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179679 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.179781 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179805 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.180352 (+ 547us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.180430 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260927 00:42:58.181051 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.174701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.0.98.3:42169)
0927 00:42:58.175101 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.175118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.175353 (+ 235us) server_negotiation.cc:407] Connection header received
0927 00:42:58.175435 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.175441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.175654 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.175805 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:58.176034 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.177163 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.178590 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.179366 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.179482 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.179616 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.180730 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.180844 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":91,"threads_started":1}
I20260927 00:42:58.183344 17696 tablet_service.cc:1467] Processing CreateTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 (DEFAULT_TABLE table=test-table [id=8b96488b428a40b98824491b8cefc54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:58.183456 17562 tablet_service.cc:1467] Processing CreateTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 (DEFAULT_TABLE table=test-table [id=8b96488b428a40b98824491b8cefc54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:58.183456 17830 tablet_service.cc:1467] Processing CreateTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 (DEFAULT_TABLE table=test-table [id=8b96488b428a40b98824491b8cefc54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:58.184506 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077ea33fbab4ef5a32d64239d6fe823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:58.184518 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077ea33fbab4ef5a32d64239d6fe823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:58.184518 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077ea33fbab4ef5a32d64239d6fe823. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:58.191288 17913 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Bootstrap starting.
I20260927 00:42:58.192826 17915 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: Bootstrap starting.
I20260927 00:42:58.194486 17913 tablet_bootstrap.cc:654] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:58.194689 17915 tablet_bootstrap.cc:654] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:58.192500 17914 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Bootstrap starting.
I20260927 00:42:58.195456 17913 log.cc:824] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:58.196578 17915 log.cc:824] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:58.197258 17914 tablet_bootstrap.cc:654] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:58.198037 17913 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: No bootstrap required, opened a new log
I20260927 00:42:58.198257 17913 ts_tablet_manager.cc:1397] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:42:58.198311 17914 log.cc:824] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:58.198578 17915 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: No bootstrap required, opened a new log
I20260927 00:42:58.198767 17915 ts_tablet_manager.cc:1397] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:42:58.200016 17914 tablet_bootstrap.cc:492] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: No bootstrap required, opened a new log
I20260927 00:42:58.200223 17914 ts_tablet_manager.cc:1397] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 00:42:58.202649 17913 raft_consensus.cc:348] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.203218 17913 raft_consensus.cc:374] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:58.203328 17913 raft_consensus.cc:729] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a260215deb6942179efe6fa4b16fc636, State: Initialized, Role: FOLLOWER
I20260927 00:42:58.203289 17914 raft_consensus.cc:348] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.203572 17914 raft_consensus.cc:374] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:58.203418 17915 raft_consensus.cc:348] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.203648 17914 raft_consensus.cc:729] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618971d7b2854781bffd0fe584fa929f, State: Initialized, Role: FOLLOWER
I20260927 00:42:58.203706 17915 raft_consensus.cc:374] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:58.203766 17915 raft_consensus.cc:729] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5358fa8870004f6d8a12dcfe5f07f931, State: Initialized, Role: FOLLOWER
I20260927 00:42:58.203946 17913 consensus_queue.cc:260] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.204101 17914 consensus_queue.cc:260] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.204175 17915 consensus_queue.cc:260] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.204849 17896 heartbeater.cc:503] Master 127.0.98.62:41281 was elected leader, sending a full tablet report...
I20260927 00:42:58.205102 17914 ts_tablet_manager.cc:1428] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:42:58.205132 17915 ts_tablet_manager.cc:1428] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:42:58.205574 17628 heartbeater.cc:503] Master 127.0.98.62:41281 was elected leader, sending a full tablet report...
I20260927 00:42:58.205811 17913 ts_tablet_manager.cc:1428] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:42:58.206193 17762 heartbeater.cc:503] Master 127.0.98.62:41281 was elected leader, sending a full tablet report...
I20260927 00:42:58.224726 17921 raft_consensus.cc:482] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:58.224967 17921 raft_consensus.cc:504] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.226243 17921 leader_election.cc:288] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a260215deb6942179efe6fa4b16fc636 (127.0.98.2:35655), 618971d7b2854781bffd0fe584fa929f (127.0.98.3:42169)
I20260927 00:42:58.230944 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.226655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33129 (local address 127.0.98.3:42169)
0927 00:42:58.226804 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.226811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.227089 (+ 278us) server_negotiation.cc:407] Connection header received
0927 00:42:58.227256 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.227259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.227308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.227367 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:58.227745 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.228357 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.229671 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.230251 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.230370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.230446 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.230706 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.230802 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:42:58.230934 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.226691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40635 (local address 127.0.98.2:35655)
0927 00:42:58.226816 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.226820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.227090 (+ 270us) server_negotiation.cc:407] Connection header received
0927 00:42:58.227281 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.227283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.227328 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.227407 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:58.227858 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.228499 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.229644 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.230264 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.230394 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.230462 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.230736 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.230816 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:42:58.230963 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.226578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42169 (local address 127.0.98.1:33129)
0927 00:42:58.226966 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.226981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.227143 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.227403 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.227406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.227423 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:58.227615 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.227622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.228471 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.228475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.229523 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.229534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.230430 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.230434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.230559 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.230582 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.230657 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.230742 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":38,"spinlock_wait_cycles":11904,"thread_start_us":136,"threads_started":1}
I20260927 00:42:58.230974 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.226629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35655 (local address 127.0.98.1:40635)
0927 00:42:58.226979 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.226991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.227140 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.227542 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.227545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.227562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:58.227735 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.227743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.228615 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.228618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.229505 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.229515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.230431 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.230436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.230595 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.230616 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.230704 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.230798 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":78,"spinlock_wait_cycles":36992,"thread_start_us":126,"threads_started":1}
I20260927 00:42:58.231729 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077ea33fbab4ef5a32d64239d6fe823" candidate_uuid: "5358fa8870004f6d8a12dcfe5f07f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618971d7b2854781bffd0fe584fa929f" is_pre_election: true
I20260927 00:42:58.231842 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077ea33fbab4ef5a32d64239d6fe823" candidate_uuid: "5358fa8870004f6d8a12dcfe5f07f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a260215deb6942179efe6fa4b16fc636" is_pre_election: true
I20260927 00:42:58.232084 17850 raft_consensus.cc:2454] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5358fa8870004f6d8a12dcfe5f07f931 in term 0.
I20260927 00:42:58.232132 17716 raft_consensus.cc:2454] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5358fa8870004f6d8a12dcfe5f07f931 in term 0.
I20260927 00:42:58.232678 17515 leader_election.cc:302] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5358fa8870004f6d8a12dcfe5f07f931, 618971d7b2854781bffd0fe584fa929f; no voters:
I20260927 00:42:58.232966 17921 raft_consensus.cc:2781] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:42:58.233059 17921 raft_consensus.cc:482] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:42:58.233107 17921 raft_consensus.cc:3037] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:58.234179 17921 raft_consensus.cc:504] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.234555 17921 leader_election.cc:288] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [CANDIDATE]: Term 1 election: Requested vote from peers a260215deb6942179efe6fa4b16fc636 (127.0.98.2:35655), 618971d7b2854781bffd0fe584fa929f (127.0.98.3:42169)
I20260927 00:42:58.235033 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077ea33fbab4ef5a32d64239d6fe823" candidate_uuid: "5358fa8870004f6d8a12dcfe5f07f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a260215deb6942179efe6fa4b16fc636"
I20260927 00:42:58.235033 17850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077ea33fbab4ef5a32d64239d6fe823" candidate_uuid: "5358fa8870004f6d8a12dcfe5f07f931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618971d7b2854781bffd0fe584fa929f"
I20260927 00:42:58.235199 17716 raft_consensus.cc:3037] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:58.235193 17850 raft_consensus.cc:3037] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:58.236658 17716 raft_consensus.cc:2454] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5358fa8870004f6d8a12dcfe5f07f931 in term 1.
I20260927 00:42:58.236658 17850 raft_consensus.cc:2454] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5358fa8870004f6d8a12dcfe5f07f931 in term 1.
I20260927 00:42:58.237063 17515 leader_election.cc:302] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5358fa8870004f6d8a12dcfe5f07f931, 618971d7b2854781bffd0fe584fa929f; no voters:
I20260927 00:42:58.237304 17921 raft_consensus.cc:2781] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:58.237594 17921 raft_consensus.cc:686] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [term 1 LEADER]: Becoming Leader. State: Replica: 5358fa8870004f6d8a12dcfe5f07f931, State: Running, Role: LEADER
I20260927 00:42:58.238174 17921 consensus_queue.cc:237] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } }
I20260927 00:42:58.242149 17434 catalog_manager.cc:5495] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5358fa8870004f6d8a12dcfe5f07f931 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5358fa8870004f6d8a12dcfe5f07f931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5358fa8870004f6d8a12dcfe5f07f931" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35843 } health_report { overall_health: HEALTHY } } }
I20260927 00:42:58.260349 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.256416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.0.98.1:35843)
0927 00:42:58.256540 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.256543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.256556 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:58.256663 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.256666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.256722 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.256877 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:42:58.257137 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.257672 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.258637 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.258902 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.259009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.259328 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:42:58.259782 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:42:58.259884 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.260023 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.260092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:42:58.266888 17716 raft_consensus.cc:1261] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 1 FOLLOWER]: Refusing update from remote peer 5358fa8870004f6d8a12dcfe5f07f931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:58.266891 17850 raft_consensus.cc:1261] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 1 FOLLOWER]: Refusing update from remote peer 5358fa8870004f6d8a12dcfe5f07f931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:58.267652 17921 consensus_queue.cc:1035] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a260215deb6942179efe6fa4b16fc636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:58.267944 17924 consensus_queue.cc:1035] T 7077ea33fbab4ef5a32d64239d6fe823 P 5358fa8870004f6d8a12dcfe5f07f931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618971d7b2854781bffd0fe584fa929f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:58.277037 17929 mvcc.cc:205] Tried to move back new op lower bound from 7333764211773517824 to 7333764211666522112. Current Snapshot: MvccSnapshot[applied={T|T < 7333764211773517824}]
I20260927 00:42:58.288183 17434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:58.288473 17434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:58.291128 17434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a65ab18d7de8400bbc42ec3462cf58cb: Sending DeleteTablet for 3 replicas of tablet 7077ea33fbab4ef5a32d64239d6fe823
I20260927 00:42:58.292155 17830 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37410
I20260927 00:42:58.292742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17499
I20260927 00:42:58.292730 17562 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43860
I20260927 00:42:58.293327 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077ea33fbab4ef5a32d64239d6fe823 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42350
I20260927 00:42:58.293561 17937 tablet_replica.cc:330] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: stopping tablet replica
I20260927 00:42:58.293807 17938 tablet_replica.cc:330] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: stopping tablet replica
I20260927 00:42:58.293958 17937 raft_consensus.cc:2229] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:58.294046 17938 raft_consensus.cc:2229] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:58.294411 17938 raft_consensus.cc:2258] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:58.294421 17937 raft_consensus.cc:2258] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:58.296531 17938 ts_tablet_manager.cc:1905] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:58.296530 17937 ts_tablet_manager.cc:1905] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:42:58.299067 17423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35843 (error 108)
I20260927 00:42:58.299477 17937 ts_tablet_manager.cc:1918] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:58.299476 17938 ts_tablet_manager.cc:1918] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:58.299590 17938 log.cc:1196] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/7077ea33fbab4ef5a32d64239d6fe823
I20260927 00:42:58.299590 17937 log.cc:1196] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/7077ea33fbab4ef5a32d64239d6fe823
I20260927 00:42:58.299943 17937 ts_tablet_manager.cc:1939] T 7077ea33fbab4ef5a32d64239d6fe823 P 618971d7b2854781bffd0fe584fa929f: Deleting consensus metadata
I20260927 00:42:58.299965 17938 ts_tablet_manager.cc:1939] T 7077ea33fbab4ef5a32d64239d6fe823 P a260215deb6942179efe6fa4b16fc636: Deleting consensus metadata
I20260927 00:42:58.299962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17633
I20260927 00:42:58.300680 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.300102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35843 (local address 127.0.0.1:43882)
0927 00:42:58.300507 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.300580 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
W20260927 00:42:58.301041 17423 catalog_manager.cc:4584] TS 5358fa8870004f6d8a12dcfe5f07f931 (127.0.98.1:35843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7077ea33fbab4ef5a32d64239d6fe823: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35843: connect: Connection refused (error 111)
I20260927 00:42:58.302161 17421 catalog_manager.cc:4841] TS 618971d7b2854781bffd0fe584fa929f (127.0.98.3:42169): tablet 7077ea33fbab4ef5a32d64239d6fe823 (table test-table [id=8b96488b428a40b98824491b8cefc54f]) successfully deleted
I20260927 00:42:58.305558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17767
I20260927 00:42:58.306694 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.306326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35655 (local address 127.0.0.1:42354)
0927 00:42:58.306542 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.306596 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 00:42:58.310838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17405
2026-09-27T00:42:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2227 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:58Z Disabled control of system clock
I20260927 00:42:58.340723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34147
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:42:58.452967 17943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:58.453250 17943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:58.453286 17943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:58.457480 17943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:58.457551 17943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:58.457566 17943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:58.457581 17943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:58.457595 17943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:58.462675 17943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:58.463755 17943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:58.465113 17943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:42:58.474129 17943 server_base.cc:1034] running on GCE node
W20260927 00:42:58.474092 17949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.474148 17951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.474097 17948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:58.474934 17943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:58.476192 17943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:58.477412 17943 hybrid_clock.cc:648] HybridClock initialized: now 1790469778477386 us; error 43 us; skew 500 ppm
I20260927 00:42:58.479370 17943 webserver.cc:460] Webserver started at http://127.0.98.62:36015/ using document root <none> and password file <none>
I20260927 00:42:58.479939 17943 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:58.480010 17943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:58.480252 17943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:58.482327 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "5ca6226492654a02b5db3b7607486a29"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "f760c81bc8fd7aa7a6aa36b1b505ab2d"
server_key_iv: "f27775543fdc3ca489c5edf294c74c1d"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.482883 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ca6226492654a02b5db3b7607486a29"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "f760c81bc8fd7aa7a6aa36b1b505ab2d"
server_key_iv: "f27775543fdc3ca489c5edf294c74c1d"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.486788 17943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:42:58.489346 17957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.490883 17943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:58.491057 17943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "5ca6226492654a02b5db3b7607486a29"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "f760c81bc8fd7aa7a6aa36b1b505ab2d"
server_key_iv: "f27775543fdc3ca489c5edf294c74c1d"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.491179 17943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:58.510556 17943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:58.511335 17943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:58.511526 17943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:58.518854 17943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34147
I20260927 00:42:58.518909 18009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34147 every 8 connection(s)
I20260927 00:42:58.519974 17943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:58.523504 18010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:58.527209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 17943
I20260927 00:42:58.527359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:42:58.527652 392 external_mini_cluster.cc:1428] Setting key dd4ae231e2d7508d8c801c9b9f2f8107
I20260927 00:42:58.530332 18010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: Bootstrap starting.
I20260927 00:42:58.532769 18010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:58.533560 18010 log.cc:824] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:58.535596 18010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: No bootstrap required, opened a new log
I20260927 00:42:58.537817 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.529445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.98.62:34147)
0927 00:42:58.530118 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.530133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.530171 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:42:58.530872 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.530892 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.531314 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.531656 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:58.532134 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.533245 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.534178 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.534687 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.534840 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.535669 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:58.535674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:58.535690 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:58.536318 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:58.536391 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:58.536416 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:58.536519 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.536663 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.537023 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260927 00:42:58.539677 18010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } }
I20260927 00:42:58.539943 18010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:58.540010 18010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca6226492654a02b5db3b7607486a29, State: Initialized, Role: FOLLOWER
I20260927 00:42:58.540498 18010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } }
I20260927 00:42:58.540668 18010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:58.540733 18010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:58.540827 18010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:58.541810 18010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } }
I20260927 00:42:58.542196 18010 leader_election.cc:302] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca6226492654a02b5db3b7607486a29; no voters:
I20260927 00:42:58.542555 18010 leader_election.cc:288] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:42:58.542637 18015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:58.542810 18015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca6226492654a02b5db3b7607486a29, State: Running, Role: LEADER
I20260927 00:42:58.543133 18015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } }
I20260927 00:42:58.543964 18010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:58.545114 18016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca6226492654a02b5db3b7607486a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } } }
I20260927 00:42:58.545267 18016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:58.545522 18017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca6226492654a02b5db3b7607486a29. Latest consensus state: current_term: 1 leader_uuid: "5ca6226492654a02b5db3b7607486a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca6226492654a02b5db3b7607486a29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34147 } } }
I20260927 00:42:58.545624 18017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:58.545832 18024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:58.549220 18024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:58.555825 18024 catalog_manager.cc:1347] Generated new cluster ID: e2ef39a35c3f47e182b992c6e791a36c
I20260927 00:42:58.555905 18024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:58.566839 18024 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:42:58.567936 18024 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:58.576259 18024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: Generated new TSK 0
I20260927 00:42:58.577210 18024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:42:58.591073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:42:58.751473 18034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:58.751760 18034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:58.751808 18034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:58.756037 18034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:58.756127 18034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:58.756232 18034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:58.761242 18034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:58.762337 18034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:58.763672 18034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:58.773502 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.773502 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:58.773705 18034 server_base.cc:1034] running on GCE node
W20260927 00:42:58.773630 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:58.774324 18034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:58.775033 18034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:58.776238 18034 hybrid_clock.cc:648] HybridClock initialized: now 1790469778776178 us; error 47 us; skew 500 ppm
I20260927 00:42:58.778524 18034 webserver.cc:460] Webserver started at http://127.0.98.1:34875/ using document root <none> and password file <none>
I20260927 00:42:58.779189 18034 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:58.779266 18034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:58.779490 18034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:58.781840 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "857d6fad16e846d792642b101e282f80"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "e9ddf237e5489c5661f992033613007f"
server_key_iv: "01981806d1fd84f318b2ccb2c58e7ec1"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.782457 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "857d6fad16e846d792642b101e282f80"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "e9ddf237e5489c5661f992033613007f"
server_key_iv: "01981806d1fd84f318b2ccb2c58e7ec1"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.786557 18034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:42:58.789202 18048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.790589 18034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:42:58.790727 18034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "857d6fad16e846d792642b101e282f80"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "e9ddf237e5489c5661f992033613007f"
server_key_iv: "01981806d1fd84f318b2ccb2c58e7ec1"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.790892 18034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:58.803175 18034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:58.804126 18034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:58.804431 18034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:58.805209 18034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:58.806273 18034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:58.806327 18034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.806373 18034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:58.806391 18034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.816109 18034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40857
I20260927 00:42:58.816141 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40857 every 8 connection(s)
I20260927 00:42:58.817168 18034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:58.819355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18034
I20260927 00:42:58.819478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:58.819775 392 external_mini_cluster.cc:1428] Setting key c3f7d81dcf62b67c4bd3b8291c392a55
I20260927 00:42:58.821985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:42:58.827296 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.819084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51183 (local address 127.0.98.62:34147)
0927 00:42:58.819217 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:58.819221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:58.820756 (+ 1535us) server_negotiation.cc:407] Connection header received
0927 00:42:58.822444 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:58.822447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:58.822504 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:58.822575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:58.823516 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.824054 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.825287 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.825525 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.825612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.826779 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:58.826782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:58.826785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:58.826828 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:58.826849 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:58.826858 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:58.826920 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:58.827072 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:42:58.827142 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:42:58.828364 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:58.819588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34147 (local address 127.0.98.1:51183)
0927 00:42:58.820579 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:58.820639 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:58.822207 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:58.822766 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:58.822773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:58.822867 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:58.823379 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.823392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.824177 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.824180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.825166 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:58.825176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:58.825669 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:58.825673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:58.825842 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:58.826484 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:58.826486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:58.826530 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:58.826608 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:58.826939 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:58.826948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:58.826953 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:42:58.827066 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:42:58.827483 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":157,"threads_started":1}
I20260927 00:42:58.829936 18162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34147
I20260927 00:42:58.830199 18162 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:58.830765 18162 heartbeater.cc:511] Master 127.0.98.62:34147 requested a full tablet report, sending...
I20260927 00:42:58.832541 17974 ts_manager.cc:194] Registered new tserver with Master: 857d6fad16e846d792642b101e282f80 (127.0.98.1:40857)
I20260927 00:42:58.834187 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51183
W20260927 00:42:58.936270 18165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:58.936569 18165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:58.936609 18165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:58.940877 18165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:58.940951 18165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:58.941009 18165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:58.946157 18165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:58.947299 18165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:58.948736 18165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:58.958719 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.958729 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:58.958729 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:58.958950 18165 server_base.cc:1034] running on GCE node
I20260927 00:42:58.959560 18165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:58.960419 18165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:58.961604 18165 hybrid_clock.cc:648] HybridClock initialized: now 1790469778961580 us; error 45 us; skew 500 ppm
I20260927 00:42:58.963717 18165 webserver.cc:460] Webserver started at http://127.0.98.2:46169/ using document root <none> and password file <none>
I20260927 00:42:58.964359 18165 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:58.964449 18165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:58.964668 18165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:58.966805 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "14a162c532f445b8a95ac6e57b7369c9"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "87b807bf4478180425b050b4c20fd96f"
server_key_iv: "d40afa532a5e9191baaaa15dda377760"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.967378 18165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14a162c532f445b8a95ac6e57b7369c9"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "87b807bf4478180425b050b4c20fd96f"
server_key_iv: "d40afa532a5e9191baaaa15dda377760"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.971527 18165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:42:58.974123 18179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.975380 18165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:42:58.975554 18165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "14a162c532f445b8a95ac6e57b7369c9"
format_stamp: "Formatted at 2026-09-27 00:42:58 on dist-test-slave-xltq"
server_key: "87b807bf4478180425b050b4c20fd96f"
server_key_iv: "d40afa532a5e9191baaaa15dda377760"
server_key_version: "encryptionkey@0"
I20260927 00:42:58.975668 18165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:58.985868 18165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:58.986598 18165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:58.986781 18165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:58.987376 18165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:58.988386 18165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:58.988437 18165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.988492 18165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:58.988516 18165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:58.998145 18165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41895
I20260927 00:42:58.998194 18292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41895 every 8 connection(s)
I20260927 00:42:58.999102 18165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:59.008409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18165
I20260927 00:42:59.008544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:59.008814 392 external_mini_cluster.cc:1428] Setting key ad922d956e52322e0f9a7a9ee825f345
I20260927 00:42:59.009159 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.001042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41843 (local address 127.0.98.62:34147)
0927 00:42:59.001200 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.001205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.002334 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 00:42:59.003826 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.003829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.003888 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.003971 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:59.005090 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.005625 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.006844 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.007082 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.007170 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.008444 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:59.008449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:59.008462 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:59.008481 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:59.008507 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:59.008518 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:59.008601 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.008928 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.009010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:42:59.010128 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.001480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34147 (local address 127.0.98.2:41843)
0927 00:42:59.002155 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.002198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.003587 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.004146 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.004154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.004293 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:59.004905 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.004921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.005756 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.005759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.006709 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.006720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.007234 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.007237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.007452 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.008106 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:59.008108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:59.008153 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:59.008270 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:59.008764 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:59.008773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:59.008778 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.008906 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.009178 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":107,"threads_started":1}
I20260927 00:42:59.010809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--builtin_ntp_servers=127.0.98.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:42:59.011359 18293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34147
I20260927 00:42:59.011588 18293 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:59.012097 18293 heartbeater.cc:511] Master 127.0.98.62:34147 requested a full tablet report, sending...
I20260927 00:42:59.013166 17974 ts_manager.cc:194] Registered new tserver with Master: 14a162c532f445b8a95ac6e57b7369c9 (127.0.98.2:41895)
I20260927 00:42:59.013731 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41843
W20260927 00:42:59.124305 18296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:59.124593 18296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:59.124632 18296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:59.128767 18296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:59.128832 18296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:59.128891 18296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:59.133934 18296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:42:59.135012 18296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:59.136358 18296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:59.146106 18301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:59.146157 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:59.146165 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:59.146214 18296 server_base.cc:1034] running on GCE node
I20260927 00:42:59.146836 18296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:59.147526 18296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:59.148698 18296 hybrid_clock.cc:648] HybridClock initialized: now 1790469779148666 us; error 45 us; skew 500 ppm
I20260927 00:42:59.150678 18296 webserver.cc:460] Webserver started at http://127.0.98.3:40215/ using document root <none> and password file <none>
I20260927 00:42:59.151304 18296 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:59.151381 18296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:59.151598 18296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:59.153728 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "06c1e13c8a2e42ed8a3f21d932e8002a"
format_stamp: "Formatted at 2026-09-27 00:42:59 on dist-test-slave-xltq"
server_key: "48be503c9f85e4d84adf2184e17c4c0a"
server_key_iv: "ade7c35cb772644a2cb386aaa76dc1ff"
server_key_version: "encryptionkey@0"
I20260927 00:42:59.154300 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "06c1e13c8a2e42ed8a3f21d932e8002a"
format_stamp: "Formatted at 2026-09-27 00:42:59 on dist-test-slave-xltq"
server_key: "48be503c9f85e4d84adf2184e17c4c0a"
server_key_iv: "ade7c35cb772644a2cb386aaa76dc1ff"
server_key_version: "encryptionkey@0"
I20260927 00:42:59.158520 18296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:42:59.161244 18310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:59.162585 18296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:59.162694 18296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "06c1e13c8a2e42ed8a3f21d932e8002a"
format_stamp: "Formatted at 2026-09-27 00:42:59 on dist-test-slave-xltq"
server_key: "48be503c9f85e4d84adf2184e17c4c0a"
server_key_iv: "ade7c35cb772644a2cb386aaa76dc1ff"
server_key_version: "encryptionkey@0"
I20260927 00:42:59.162798 18296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:59.183676 18296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:59.184435 18296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:59.184599 18296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:59.185155 18296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:59.186100 18296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:59.186146 18296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:59.186187 18296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:59.186205 18296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:59.194680 18296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46073
I20260927 00:42:59.194715 18423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46073 every 8 connection(s)
I20260927 00:42:59.195699 18296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:59.197659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18296
I20260927 00:42:59.197762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:59.198007 392 external_mini_cluster.cc:1428] Setting key 62947a16b5afcef260f50baecb566620
I20260927 00:42:59.206027 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.197602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33747 (local address 127.0.98.62:34147)
0927 00:42:59.197804 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.197810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.199100 (+ 1290us) server_negotiation.cc:407] Connection header received
0927 00:42:59.200831 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.200834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.200882 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.200967 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:59.202025 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.202581 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.203862 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.204089 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.204184 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.205538 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:59.205541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:59.205544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:59.205559 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:59.205581 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:59.205589 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:59.205652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.205838 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.205902 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:42:59.206995 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.198063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34147 (local address 127.0.98.3:33747)
0927 00:42:59.198939 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.199002 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.200616 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.201185 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.201193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.201286 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:59.201875 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.201890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.202699 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.202702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.203695 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.203706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.204286 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.204291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.204511 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.205181 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:59.205183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:59.205230 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:59.205307 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:59.205659 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:59.205664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:59.205668 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.205763 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.206030 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":169,"threads_started":1}
I20260927 00:42:59.208357 18424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34147
I20260927 00:42:59.208627 18424 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:59.209257 18424 heartbeater.cc:511] Master 127.0.98.62:34147 requested a full tablet report, sending...
I20260927 00:42:59.210469 17974 ts_manager.cc:194] Registered new tserver with Master: 06c1e13c8a2e42ed8a3f21d932e8002a (127.0.98.3:46073)
I20260927 00:42:59.211171 17974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33747
I20260927 00:42:59.212393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:42:59.217846 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.214433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.62:34147)
0927 00:42:59.214587 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.214591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.214764 (+ 173us) server_negotiation.cc:407] Connection header received
0927 00:42:59.214845 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.214847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.214889 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.214954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:59.215173 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.215673 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.216558 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.216763 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.216839 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.217532 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:59.217535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:59.217538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:59.217550 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:59.217568 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:59.217576 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:59.217627 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.217678 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.217726 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:42:59.220709 17974 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:42:59.223641 17974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:42:59.238793 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.232848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40857 (local address 127.0.0.1:55024)
0927 00:42:59.233531 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.233572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.233715 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.234377 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.234383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.234407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:59.234569 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.234576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.235909 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.235912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.237375 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.237384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.238373 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.238375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.238455 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.238489 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.238566 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.238666 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":114,"threads_started":1}
I20260927 00:42:59.238934 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.233008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.0.98.1:40857)
0927 00:42:59.233351 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.233363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.233874 (+ 511us) server_negotiation.cc:407] Connection header received
0927 00:42:59.233962 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.233980 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.234220 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.234388 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:59.234702 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.235800 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.237537 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.238246 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.238379 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.238599 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.238654 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.238760 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20260927 00:42:59.239653 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.232999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41895 (local address 127.0.0.1:36348)
0927 00:42:59.234698 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.234710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.234867 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.235385 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.235388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.235402 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:59.235579 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.235586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.236670 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.236673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.238061 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.238072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.239268 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.239271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.239344 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.239365 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.239433 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.239515 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":110,"thread_start_us":80,"threads_started":1}
I20260927 00:42:59.239744 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.233709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.98.2:41895)
0927 00:42:59.234131 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.234146 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.234793 (+ 647us) server_negotiation.cc:407] Connection header received
0927 00:42:59.235033 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.235039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.235224 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.235365 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:59.235691 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.236559 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.238207 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.238986 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.239088 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.239274 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.239498 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.239600 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":79,"threads_started":1}
I20260927 00:42:59.241050 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.233833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.0.98.3:46073)
0927 00:42:59.234260 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.234273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.234603 (+ 330us) server_negotiation.cc:407] Connection header received
0927 00:42:59.234914 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.234921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.235102 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.235229 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:59.235478 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.236613 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.239347 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.240121 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.240266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.240742 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.240794 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.240882 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260927 00:42:59.241236 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.233649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46073 (local address 127.0.0.1:47608)
0927 00:42:59.234488 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.234504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.234605 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.235223 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.235227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.235242 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:59.235351 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.235357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.236724 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.236727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.238066 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.238072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.240272 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.240275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.240363 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.240387 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.240444 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.240511 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":78,"thread_start_us":89,"threads_started":1}
I20260927 00:42:59.242101 18096 tablet_service.cc:1467] Processing CreateTablet for tablet 8002c1410c8b46d09539e027f78131b6 (DEFAULT_TABLE table=test-table [id=4092c6a051634492bf45d581cb178a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:59.242449 18227 tablet_service.cc:1467] Processing CreateTablet for tablet 8002c1410c8b46d09539e027f78131b6 (DEFAULT_TABLE table=test-table [id=4092c6a051634492bf45d581cb178a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:59.243033 18096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8002c1410c8b46d09539e027f78131b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:59.243232 18227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8002c1410c8b46d09539e027f78131b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:59.246408 18358 tablet_service.cc:1467] Processing CreateTablet for tablet 8002c1410c8b46d09539e027f78131b6 (DEFAULT_TABLE table=test-table [id=4092c6a051634492bf45d581cb178a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:59.247400 18358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8002c1410c8b46d09539e027f78131b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:59.250797 18442 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Bootstrap starting.
I20260927 00:42:59.251281 18441 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Bootstrap starting.
I20260927 00:42:59.253528 18442 tablet_bootstrap.cc:654] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:59.253628 18441 tablet_bootstrap.cc:654] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:59.254251 18442 log.cc:824] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:59.254472 18441 log.cc:824] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:59.254801 18443 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Bootstrap starting.
I20260927 00:42:59.256628 18441 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: No bootstrap required, opened a new log
I20260927 00:42:59.256640 18442 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: No bootstrap required, opened a new log
I20260927 00:42:59.256824 18442 ts_tablet_manager.cc:1397] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:42:59.256825 18441 ts_tablet_manager.cc:1397] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 00:42:59.257130 18443 tablet_bootstrap.cc:654] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:59.257994 18443 log.cc:824] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:59.260396 18443 tablet_bootstrap.cc:492] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: No bootstrap required, opened a new log
I20260927 00:42:59.260633 18443 ts_tablet_manager.cc:1397] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:42:59.260564 18441 raft_consensus.cc:348] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.260828 18441 raft_consensus.cc:374] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:59.260880 18441 raft_consensus.cc:729] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a162c532f445b8a95ac6e57b7369c9, State: Initialized, Role: FOLLOWER
I20260927 00:42:59.260937 18442 raft_consensus.cc:348] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.261220 18442 raft_consensus.cc:374] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:59.261276 18442 raft_consensus.cc:729] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857d6fad16e846d792642b101e282f80, State: Initialized, Role: FOLLOWER
I20260927 00:42:59.261332 18441 consensus_queue.cc:260] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.261685 18442 consensus_queue.cc:260] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.262079 18293 heartbeater.cc:503] Master 127.0.98.62:34147 was elected leader, sending a full tablet report...
I20260927 00:42:59.262390 18441 ts_tablet_manager.cc:1428] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:42:59.262658 18442 ts_tablet_manager.cc:1428] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:42:59.262998 18162 heartbeater.cc:503] Master 127.0.98.62:34147 was elected leader, sending a full tablet report...
I20260927 00:42:59.265365 18443 raft_consensus.cc:348] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.265681 18443 raft_consensus.cc:374] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:59.265736 18443 raft_consensus.cc:729] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c1e13c8a2e42ed8a3f21d932e8002a, State: Initialized, Role: FOLLOWER
I20260927 00:42:59.266170 18443 consensus_queue.cc:260] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.266898 18424 heartbeater.cc:503] Master 127.0.98.62:34147 was elected leader, sending a full tablet report...
I20260927 00:42:59.267212 18443 ts_tablet_manager.cc:1428] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 00:42:59.318475 18163 tablet.cc:2369] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:42:59.384485 18447 raft_consensus.cc:482] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:59.384727 18447 raft_consensus.cc:504] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.385866 18447 leader_election.cc:288] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857d6fad16e846d792642b101e282f80 (127.0.98.1:40857), 06c1e13c8a2e42ed8a3f21d932e8002a (127.0.98.3:46073)
I20260927 00:42:59.390224 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.386182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40857 (local address 127.0.98.2:59413)
0927 00:42:59.386393 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.386409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.386511 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.386846 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.386849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.386868 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:59.387041 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.387052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.387977 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.387981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.388955 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.388965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.389791 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.389795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.389896 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.389928 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.390005 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.390078 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:42:59.390236 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.386289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59413 (local address 127.0.98.1:40857)
0927 00:42:59.386448 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.386454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.386475 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:42:59.386659 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.386663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.386720 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.386810 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:59.387187 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.387868 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.389089 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.389632 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.389713 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.389766 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.390031 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.390095 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 00:42:59.390672 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.386653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34555 (local address 127.0.98.3:46073)
0927 00:42:59.386771 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.386775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.387003 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:42:59.387185 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.387189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.387236 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.387314 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:59.387636 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.388297 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.389527 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.390058 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.390147 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.390200 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.390421 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.390561 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:42:59.390977 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.386461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46073 (local address 127.0.98.2:34555)
0927 00:42:59.386923 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.386936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:59.387067 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:59.387341 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:59.387344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:59.387365 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:59.387490 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.387496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.388402 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.388404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.389372 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:59.389382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.390185 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.390189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:59.390281 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.390302 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:42:59.390803 (+ 501us) client_negotiation.cc:240] Negotiation successful
0927 00:42:59.390872 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":134,"threads_started":1}
I20260927 00:42:59.390969 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8002c1410c8b46d09539e027f78131b6" candidate_uuid: "14a162c532f445b8a95ac6e57b7369c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857d6fad16e846d792642b101e282f80" is_pre_election: true
I20260927 00:42:59.391289 18116 raft_consensus.cc:2454] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a162c532f445b8a95ac6e57b7369c9 in term 0.
I20260927 00:42:59.391570 18378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8002c1410c8b46d09539e027f78131b6" candidate_uuid: "14a162c532f445b8a95ac6e57b7369c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" is_pre_election: true
I20260927 00:42:59.391844 18378 raft_consensus.cc:2454] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14a162c532f445b8a95ac6e57b7369c9 in term 0.
I20260927 00:42:59.391820 18180 leader_election.cc:302] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a162c532f445b8a95ac6e57b7369c9, 857d6fad16e846d792642b101e282f80; no voters:
I20260927 00:42:59.392114 18447 raft_consensus.cc:2781] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:42:59.392241 18447 raft_consensus.cc:482] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:42:59.392297 18447 raft_consensus.cc:3037] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:59.393298 18447 raft_consensus.cc:504] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.393625 18447 leader_election.cc:288] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [CANDIDATE]: Term 1 election: Requested vote from peers 857d6fad16e846d792642b101e282f80 (127.0.98.1:40857), 06c1e13c8a2e42ed8a3f21d932e8002a (127.0.98.3:46073)
I20260927 00:42:59.394071 18378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8002c1410c8b46d09539e027f78131b6" candidate_uuid: "14a162c532f445b8a95ac6e57b7369c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a"
I20260927 00:42:59.394071 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8002c1410c8b46d09539e027f78131b6" candidate_uuid: "14a162c532f445b8a95ac6e57b7369c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857d6fad16e846d792642b101e282f80"
I20260927 00:42:59.394241 18116 raft_consensus.cc:3037] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:59.394222 18378 raft_consensus.cc:3037] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:59.395663 18378 raft_consensus.cc:2454] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a162c532f445b8a95ac6e57b7369c9 in term 1.
I20260927 00:42:59.395663 18116 raft_consensus.cc:2454] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14a162c532f445b8a95ac6e57b7369c9 in term 1.
I20260927 00:42:59.396021 18180 leader_election.cc:302] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14a162c532f445b8a95ac6e57b7369c9, 857d6fad16e846d792642b101e282f80; no voters:
I20260927 00:42:59.396265 18447 raft_consensus.cc:2781] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:59.396503 18447 raft_consensus.cc:686] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 1 LEADER]: Becoming Leader. State: Replica: 14a162c532f445b8a95ac6e57b7369c9, State: Running, Role: LEADER
I20260927 00:42:59.396843 18447 consensus_queue.cc:237] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } }
I20260927 00:42:59.400426 17973 catalog_manager.cc:5495] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14a162c532f445b8a95ac6e57b7369c9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14a162c532f445b8a95ac6e57b7369c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14a162c532f445b8a95ac6e57b7369c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 } health_report { overall_health: UNKNOWN } } }
I20260927 00:42:59.433640 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.429918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.0.98.2:41895)
0927 00:42:59.430035 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:59.430038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:59.430052 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:59.430138 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:59.430141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:59.430220 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:59.430320 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:42:59.430623 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.431141 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.432059 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:59.432396 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:59.432496 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:59.432809 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:42:59.433244 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:42:59.433433 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:59.433460 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 00:42:59.433510 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:42:59.440176 18378 raft_consensus.cc:1261] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 1 FOLLOWER]: Refusing update from remote peer 14a162c532f445b8a95ac6e57b7369c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:59.440989 18447 consensus_queue.cc:1035] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06c1e13c8a2e42ed8a3f21d932e8002a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:59.445830 18116 raft_consensus.cc:1261] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 1 FOLLOWER]: Refusing update from remote peer 14a162c532f445b8a95ac6e57b7369c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 00:42:59.446784 18425 tablet.cc:2369] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:42:59.446743 18447 consensus_queue.cc:1035] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857d6fad16e846d792642b101e282f80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:59.450217 18454 mvcc.cc:205] Tried to move back new op lower bound from 7333764216578564096 to 7333764216412168192. Current Snapshot: MvccSnapshot[applied={T|T < 7333764216578564096}]
I20260927 00:42:59.453537 18455 mvcc.cc:205] Tried to move back new op lower bound from 7333764216578564096 to 7333764216412168192. Current Snapshot: MvccSnapshot[applied={T|T < 7333764216578564096 or (T in {7333764216578564096})}]
I20260927 00:42:59.456257 18460 mvcc.cc:205] Tried to move back new op lower bound from 7333764216578564096 to 7333764216412168192. Current Snapshot: MvccSnapshot[applied={T|T < 7333764216578564096}]
I20260927 00:42:59.464726 17972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:59.464951 17972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:59.468128 17972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ca6226492654a02b5db3b7607486a29: Sending DeleteTablet for 3 replicas of tablet 8002c1410c8b46d09539e027f78131b6
I20260927 00:42:59.469336 18358 tablet_service.cc:1514] Processing DeleteTablet for tablet 8002c1410c8b46d09539e027f78131b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:59 UTC) from {username='slave'} at 127.0.0.1:47608
I20260927 00:42:59.469374 18096 tablet_service.cc:1514] Processing DeleteTablet for tablet 8002c1410c8b46d09539e027f78131b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:59 UTC) from {username='slave'} at 127.0.0.1:55024
I20260927 00:42:59.469911 18464 tablet_replica.cc:330] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: stopping tablet replica
I20260927 00:42:59.469933 18227 tablet_service.cc:1514] Processing DeleteTablet for tablet 8002c1410c8b46d09539e027f78131b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:59 UTC) from {username='slave'} at 127.0.0.1:36348
I20260927 00:42:59.470281 18464 raft_consensus.cc:2229] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:59.470448 18466 tablet_replica.cc:330] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: stopping tablet replica
I20260927 00:42:59.470647 18464 raft_consensus.cc:2258] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:59.471028 18465 tablet_replica.cc:330] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80: stopping tablet replica
I20260927 00:42:59.471056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18034
I20260927 00:42:59.471300 18465 raft_consensus.cc:2229] T 8002c1410c8b46d09539e027f78131b6 P 857d6fad16e846d792642b101e282f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:59.472553 18464 ts_tablet_manager.cc:1905] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:59.470820 18466 raft_consensus.cc:2229] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:42:59.473187 18466 raft_consensus.cc:2258] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:59.476478 18464 ts_tablet_manager.cc:1918] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:59.478134 18466 ts_tablet_manager.cc:1905] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:59.481103 18466 ts_tablet_manager.cc:1918] T 8002c1410c8b46d09539e027f78131b6 P 14a162c532f445b8a95ac6e57b7369c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:59.481384 18464 log.cc:1196] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/8002c1410c8b46d09539e027f78131b6
W20260927 00:42:59.481653 17958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40857 (error 108)
I20260927 00:42:59.481853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18165
I20260927 00:42:59.483351 18464 ts_tablet_manager.cc:1939] T 8002c1410c8b46d09539e027f78131b6 P 06c1e13c8a2e42ed8a3f21d932e8002a: Deleting consensus metadata
I20260927 00:42:59.483783 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.483431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40857 (local address 127.0.0.1:55040)
0927 00:42:59.483589 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.483682 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:42:59.484179 17958 catalog_manager.cc:4584] TS 857d6fad16e846d792642b101e282f80 (127.0.98.1:40857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8002c1410c8b46d09539e027f78131b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40857: connect: Connection refused (error 111)
I20260927 00:42:59.485010 17958 catalog_manager.cc:4841] TS 06c1e13c8a2e42ed8a3f21d932e8002a (127.0.98.3:46073): tablet 8002c1410c8b46d09539e027f78131b6 (table test-table [id=4092c6a051634492bf45d581cb178a2b]) successfully deleted
I20260927 00:42:59.490204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18296
I20260927 00:42:59.490437 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:59.490134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41895 (local address 127.0.0.1:36376)
0927 00:42:59.490291 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:59.490346 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:42:59.495633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 17943
2026-09-27T00:42:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:42:59 dist-test-slave-xltq krb5kdc[18470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:42:59 dist-test-slave-xltq krb5kdc[18470](info): set up 2 sockets
Sep 27 00:42:59 dist-test-slave-xltq krb5kdc[18470](info): commencing operation
krb5kdc: starting...
W20260927 00:43:00.464044 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:00 dist-test-slave-xltq krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:00.588032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:00.701126 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:00.701427 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:00.701468 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:00.705739 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:00.705821 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:00.705839 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:00.705855 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:00.705869 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:00.711702 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:00.713008 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:00.714128 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:00.723169 18491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:00.723182 18489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:00.723182 18488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:00.723392 18483 server_base.cc:1034] running on GCE node
I20260927 00:43:00.724236 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:00.725625 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:00.726830 18483 hybrid_clock.cc:648] HybridClock initialized: now 1790469780726808 us; error 43 us; skew 500 ppm
Sep 27 00:43:00 dist-test-slave-xltq krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:00.730152 18483 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:00.731243 18483 webserver.cc:460] Webserver started at http://127.0.98.62:41947/ using document root <none> and password file <none>
I20260927 00:43:00.731825 18483 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:00.731895 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:00.732090 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:00.734376 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "d719209eae7f46d5b33d752ffe07f10a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "580e71de08b73abad28ae291595b13a8"
server_key_iv: "1dd486e31d73d9e64426d2e7c9f76a14"
server_key_version: "encryptionkey@0"
I20260927 00:43:00.734988 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "d719209eae7f46d5b33d752ffe07f10a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "580e71de08b73abad28ae291595b13a8"
server_key_iv: "1dd486e31d73d9e64426d2e7c9f76a14"
server_key_version: "encryptionkey@0"
I20260927 00:43:00.739012 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:00.741634 18498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:00.743007 18483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:43:00.743139 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "d719209eae7f46d5b33d752ffe07f10a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "580e71de08b73abad28ae291595b13a8"
server_key_iv: "1dd486e31d73d9e64426d2e7c9f76a14"
server_key_version: "encryptionkey@0"
I20260927 00:43:00.743253 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:00.754407 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:00.757907 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:00.758097 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:00.764957 18483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44243
I20260927 00:43:00.765080 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44243 every 8 connection(s)
I20260927 00:43:00.765992 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:00.769044 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:00.774994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18483
I20260927 00:43:00.775139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:00.775132 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: Bootstrap starting.
I20260927 00:43:00.775431 392 external_mini_cluster.cc:1428] Setting key 72245bf4229d1090f8a0c8bb73713982
I20260927 00:43:00.777825 18551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:00.778595 18551 log.cc:824] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:00.780829 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: No bootstrap required, opened a new log
Sep 27 00:43:00 dist-test-slave-xltq krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:00.784602 18551 raft_consensus.cc:348] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 00:43:00.784934 18551 raft_consensus.cc:374] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:00.785006 18551 raft_consensus.cc:729] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d719209eae7f46d5b33d752ffe07f10a, State: Initialized, Role: FOLLOWER
I20260927 00:43:00.785532 18551 consensus_queue.cc:260] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 00:43:00.785710 18551 raft_consensus.cc:388] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:00.785780 18551 raft_consensus.cc:482] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:00.785881 18551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:00.786926 18551 raft_consensus.cc:504] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 00:43:00.787451 18551 leader_election.cc:302] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d719209eae7f46d5b33d752ffe07f10a; no voters:
I20260927 00:43:00.787884 18551 leader_election.cc:288] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:00.787992 18556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:00.788161 18556 raft_consensus.cc:686] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [term 1 LEADER]: Becoming Leader. State: Replica: d719209eae7f46d5b33d752ffe07f10a, State: Running, Role: LEADER
I20260927 00:43:00.788538 18556 consensus_queue.cc:237] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } }
I20260927 00:43:00.789376 18551 sys_catalog.cc:564] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:00.790508 18558 sys_catalog.cc:455] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d719209eae7f46d5b33d752ffe07f10a. Latest consensus state: current_term: 1 leader_uuid: "d719209eae7f46d5b33d752ffe07f10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260927 00:43:00.790683 18558 sys_catalog.cc:458] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:00.790489 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d719209eae7f46d5b33d752ffe07f10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d719209eae7f46d5b33d752ffe07f10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44243 } } }
I20260927 00:43:00.790819 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:00.790867 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:00.776956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.0.98.62:44243)
0927 00:43:00.777430 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:00.777446 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:00.777483 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:00.778283 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:00.778303 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:00.778684 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:00.779031 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:00.779805 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:00.780956 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:00.781820 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:00.782324 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:00.782493 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:00.785117 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:00.785122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:00.785137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:00.785172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:00.786878 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:00.787641 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:00.787648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:00.787655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:00.787790 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:00.788135 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:00.788139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:00.788141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:00.788806 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:00.789092 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:00.789361 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 00:43:00.789724 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260927 00:43:00.791158 18565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:00.794490 18565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:00.800302 18565 catalog_manager.cc:1347] Generated new cluster ID: 0bc2ef1eafdb4d9ea6e8cf3003be339e
I20260927 00:43:00.800427 18565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:00.808408 18565 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:00.810086 18565 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:00.822691 18565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: Generated new TSK 0
I20260927 00:43:00.823650 18565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:43:00.855722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:00.972329 18577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:00.972620 18577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:00.972664 18577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:00.976861 18577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:00.976933 18577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:00.977000 18577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:00.982625 18577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:00.983786 18577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:00.984851 18577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:00.995846 18582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:00.995865 18583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:00.995873 18577 server_base.cc:1034] running on GCE node
W20260927 00:43:00.995879 18585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:00.996654 18577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:00.997388 18577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:00.998587 18577 hybrid_clock.cc:648] HybridClock initialized: now 1790469780998560 us; error 44 us; skew 500 ppm
Sep 27 00:43:00 dist-test-slave-xltq krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:01.001756 18577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:01.002756 18577 webserver.cc:460] Webserver started at http://127.0.98.1:42695/ using document root <none> and password file <none>
I20260927 00:43:01.003362 18577 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:01.003448 18577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:01.003669 18577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:01.005996 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "64f711e2f5844f709b2ae3b9dff2692a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "21aedf376c7c2104a4ed5880adb1db4b"
server_key_iv: "2b4c4fa86b6c167ce4ee2ebc917f5788"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.006630 18577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "64f711e2f5844f709b2ae3b9dff2692a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "21aedf376c7c2104a4ed5880adb1db4b"
server_key_iv: "2b4c4fa86b6c167ce4ee2ebc917f5788"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.011085 18577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:43:01.014083 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.015590 18577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:01.015738 18577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "64f711e2f5844f709b2ae3b9dff2692a"
format_stamp: "Formatted at 2026-09-27 00:43:00 on dist-test-slave-xltq"
server_key: "21aedf376c7c2104a4ed5880adb1db4b"
server_key_iv: "2b4c4fa86b6c167ce4ee2ebc917f5788"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.015890 18577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:01.040943 18577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:01.044443 18577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:01.044620 18577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:01.045185 18577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:01.046026 18577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:01.046079 18577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.046125 18577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:01.046146 18577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.054991 18577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43471
I20260927 00:43:01.055003 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43471 every 8 connection(s)
I20260927 00:43:01.055956 18577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:01.063002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18577
I20260927 00:43:01.063140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:01.063439 392 external_mini_cluster.cc:1428] Setting key 0b84f51d46560b2e8ec772aa879bf161
Sep 27 00:43:01 dist-test-slave-xltq krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:01.072041 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.057925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41037 (local address 127.0.98.62:44243)
0927 00:43:01.058051 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.058055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.058931 (+ 876us) server_negotiation.cc:407] Connection header received
0927 00:43:01.060254 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.060258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.060312 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.060402 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:01.061657 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.062203 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.063771 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.064138 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.064274 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.067640 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:01.067643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:01.067646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:01.067688 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:01.069484 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.070170 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.070174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.070175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.070285 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.070691 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.070695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.070697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.070924 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:01.071025 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.071652 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.071838 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:43:01.072983 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.058251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44243 (local address 127.0.98.1:41037)
0927 00:43:01.058766 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.058813 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.059940 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.060561 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.060568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.060934 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:01.061494 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.061510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.062317 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.062320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.063602 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.063616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.064506 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.064510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.064734 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.065412 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:01.065431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:01.067396 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:01.069668 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.069675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.069692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.070041 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.070400 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.070402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.070404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.070586 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.071060 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:01.071068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:01.071375 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.071584 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.071996 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:43:01.074577 18706 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44243
I20260927 00:43:01.074879 18706 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:01.075461 18706 heartbeater.cc:511] Master 127.0.98.62:44243 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:43:01.077445 18515 ts_manager.cc:194] Registered new tserver with Master: 64f711e2f5844f709b2ae3b9dff2692a (127.0.98.1:43471)
I20260927 00:43:01.079072 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:43:01.095690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:01.207690 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:01.207994 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:01.208055 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:01.212188 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:01.212292 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:01.212379 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:01.217979 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:01.219201 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:01.220368 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:43:01.231117 18711 server_base.cc:1034] running on GCE node
W20260927 00:43:01.231029 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:01.231032 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:01.231050 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:01.231889 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:01.232746 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:01.233938 18711 hybrid_clock.cc:648] HybridClock initialized: now 1790469781233905 us; error 51 us; skew 500 ppm
Sep 27 00:43:01 dist-test-slave-xltq krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:01.241354 18711 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:01.242357 18711 webserver.cc:460] Webserver started at http://127.0.98.2:39267/ using document root <none> and password file <none>
I20260927 00:43:01.242937 18711 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:01.243006 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:01.243213 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:01.245395 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "351ff89a40c54fbc9fe91b18712fe795"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "2c97d8092c8fcafca9164c3b6e7413cb"
server_key_iv: "0e26b8d65d445946ae1dcbda87364853"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.245983 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "351ff89a40c54fbc9fe91b18712fe795"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "2c97d8092c8fcafca9164c3b6e7413cb"
server_key_iv: "0e26b8d65d445946ae1dcbda87364853"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.250114 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:01.252769 18726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.254025 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:01.254135 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "351ff89a40c54fbc9fe91b18712fe795"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "2c97d8092c8fcafca9164c3b6e7413cb"
server_key_iv: "0e26b8d65d445946ae1dcbda87364853"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.254235 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:01.266198 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:01.270051 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:01.270233 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:01.270831 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:01.271733 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:01.271785 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.271826 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:01.271842 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.280750 18711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45761
I20260927 00:43:01.280773 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45761 every 8 connection(s)
I20260927 00:43:01.281651 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:01.282486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18711
I20260927 00:43:01.282606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:01.282869 392 external_mini_cluster.cc:1428] Setting key 06bdf22306a5e0d6833c6611445e39e1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:43:01 dist-test-slave-xltq krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:43:01.297905 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.283728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60869 (local address 127.0.98.62:44243)
0927 00:43:01.283861 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.283865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.284962 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 00:43:01.286226 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.286239 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.286318 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.286432 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:01.287824 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.288505 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.290474 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.290834 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.290954 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.294076 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:01.294080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:01.294082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:01.294114 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:01.295627 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.296243 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.296246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.296248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.296337 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.296677 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.296680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.296682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.296883 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:01.296977 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.297598 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.297736 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:01.298789 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.284151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44243 (local address 127.0.98.2:60869)
0927 00:43:01.284772 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.284837 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.285903 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.286622 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.286629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.287022 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:01.287632 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.287654 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.288660 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.288663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.290285 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.290304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.291041 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.291046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.291272 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.292007 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:01.292032 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:01.293809 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:01.295764 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.295772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.295794 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.296112 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.296427 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.296430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.296432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.296588 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.296982 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:01.296990 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:01.297344 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.297537 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.297916 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":161,"threads_started":1}
I20260927 00:43:01.299921 18840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44243
I20260927 00:43:01.300163 18840 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:01.300800 18840 heartbeater.cc:511] Master 127.0.98.62:44243 requested a full tablet report, sending...
I20260927 00:43:01.301955 18515 ts_manager.cc:194] Registered new tserver with Master: 351ff89a40c54fbc9fe91b18712fe795 (127.0.98.2:45761)
I20260927 00:43:01.302670 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:43:01.311650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:01.426034 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:01.426357 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:01.426411 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:01.430730 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:01.430823 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:01.430886 18845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:01.436758 18845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:01.437918 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:01.438977 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:01.449319 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:01.449489 18845 server_base.cc:1034] running on GCE node
W20260927 00:43:01.449365 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:01.449319 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:01.450035 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:01.450699 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:01.451885 18845 hybrid_clock.cc:648] HybridClock initialized: now 1790469781451858 us; error 49 us; skew 500 ppm
Sep 27 00:43:01 dist-test-slave-xltq krb5kdc[18470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:01.454907 18845 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:01.455938 18845 webserver.cc:460] Webserver started at http://127.0.98.3:40589/ using document root <none> and password file <none>
I20260927 00:43:01.456558 18845 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:01.456640 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:01.456851 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:01.459167 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "fc7214046c2a7ff0e4305579246d6b48"
server_key_iv: "5f4ddff02fd3d57a4c736c2a847ce3c8"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.459749 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "fc7214046c2a7ff0e4305579246d6b48"
server_key_iv: "5f4ddff02fd3d57a4c736c2a847ce3c8"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.463815 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 00:43:01.466674 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.468263 18845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:01.468405 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "fc7214046c2a7ff0e4305579246d6b48"
server_key_iv: "5f4ddff02fd3d57a4c736c2a847ce3c8"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.468597 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:01.503755 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:01.507242 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:01.507442 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:01.508096 18845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:01.509055 18845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:01.509107 18845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.509164 18845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:01.509191 18845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.518234 18845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41281
I20260927 00:43:01.518309 18973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41281 every 8 connection(s)
I20260927 00:43:01.519272 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:01.528831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 18845
I20260927 00:43:01.528970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:01.529238 392 external_mini_cluster.cc:1428] Setting key d6583e2e460055dace1a7f530e474162
Sep 27 00:43:01 dist-test-slave-xltq krb5kdc[18470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:01.534706 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.521334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50467 (local address 127.0.98.62:44243)
0927 00:43:01.521505 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.521510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.522542 (+ 1032us) server_negotiation.cc:407] Connection header received
0927 00:43:01.523989 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.523992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.524048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.524126 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:01.525591 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.526140 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.527517 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.527763 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.527866 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.531050 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:01.531054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:01.531056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:01.531084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:01.532625 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.533206 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.533209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.533210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.533298 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.533663 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.533666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.533667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.533870 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:01.533949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.534457 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.534579 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:43:01.535689 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.521854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44243 (local address 127.0.98.3:50467)
0927 00:43:01.522361 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.522412 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.523732 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.524359 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.524367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.524816 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:01.525350 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.525366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.526309 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.526312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.527361 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.527372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.527962 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.527965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.528150 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.528960 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:01.528985 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:01.530825 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:01.532747 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.532754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.532770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.533090 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.533393 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:01.533401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:01.533403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:01.533569 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:01.533957 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:01.533963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:01.534232 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.534481 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.534775 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
I20260927 00:43:01.537081 18974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44243
I20260927 00:43:01.537340 18974 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:01.537909 18974 heartbeater.cc:511] Master 127.0.98.62:44243 requested a full tablet report, sending...
I20260927 00:43:01.539063 18515 ts_manager.cc:194] Registered new tserver with Master: dfd4a5899c9b4a9cb6801c04b89eadc7 (127.0.98.3:41281)
I20260927 00:43:01.539749 18515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50467
I20260927 00:43:01.543632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:01.556389 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.548054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.98.62:44243)
0927 00:43:01.548232 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.548237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.548493 (+ 256us) server_negotiation.cc:407] Connection header received
0927 00:43:01.548648 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.548651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.548702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.548772 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:01.549506 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.550037 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.550932 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.551162 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.551266 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.553090 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:01.553093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:01.553095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:01.553128 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:01.554480 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.555038 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.555041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.555042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.555131 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:01.555437 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:01.555439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:01.555441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:01.555685 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:01.555825 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.556101 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.556196 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:43:01.559520 18515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:01.562674 18515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:01.578063 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.571861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.0.1:44776)
0927 00:43:01.572279 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.572341 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.572481 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.573342 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.573349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.573375 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:01.573530 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.573537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.575058 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.575062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.576336 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.576348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.577451 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.577454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.577571 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.577608 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.577717 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.577857 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260927 00:43:01.578182 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.573219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.98.2:45761)
0927 00:43:01.573688 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.573704 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.573725 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:43:01.573792 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.573796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.573978 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.574129 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:01.574350 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.575318 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.576678 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.577519 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.577617 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.577754 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.577906 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.578018 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":150,"threads_started":1}
I20260927 00:43:01.578281 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.572066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.0.98.1:43471)
0927 00:43:01.572816 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.572831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.572856 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:43:01.572937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.572942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.573179 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.573348 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:01.573723 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.574923 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.576499 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.577333 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.577435 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.577615 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.578042 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.578145 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":149,"threads_started":1}
I20260927 00:43:01.578538 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.572767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45761 (local address 127.0.0.1:39474)
0927 00:43:01.573030 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.573045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.573138 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.574117 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.574121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.574139 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:01.574236 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.574242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.575450 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.575454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.576538 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.576549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.577694 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.577697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.577774 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.577794 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.577858 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.577911 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260927 00:43:01.578987 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.572883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41281 (local address 127.0.0.1:49324)
0927 00:43:01.573658 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.573670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.573819 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.574523 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.574524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.574538 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:01.574608 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.574611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.575867 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.575870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.577027 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.577037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.578453 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.578457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.578562 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.578583 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.578788 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.578862 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":24,"thread_start_us":84,"threads_started":1}
I20260927 00:43:01.579115 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.572499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.0.98.3:41281)
0927 00:43:01.573352 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.573362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.573752 (+ 390us) server_negotiation.cc:407] Connection header received
0927 00:43:01.574029 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.574033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.574209 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.574331 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:01.574709 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.575724 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.577178 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.577953 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.578028 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.578149 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.578848 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.578961 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":160,"threads_started":1}
I20260927 00:43:01.581498 18640 tablet_service.cc:1467] Processing CreateTablet for tablet d3510332b7744ca5861cc5bd73d1e408 (DEFAULT_TABLE table=test-table [id=19ecc6e054244bee9ca367ee6468e01b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:01.581501 18774 tablet_service.cc:1467] Processing CreateTablet for tablet d3510332b7744ca5861cc5bd73d1e408 (DEFAULT_TABLE table=test-table [id=19ecc6e054244bee9ca367ee6468e01b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:01.582677 18640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3510332b7744ca5861cc5bd73d1e408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:01.582687 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3510332b7744ca5861cc5bd73d1e408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:01.582829 18908 tablet_service.cc:1467] Processing CreateTablet for tablet d3510332b7744ca5861cc5bd73d1e408 (DEFAULT_TABLE table=test-table [id=19ecc6e054244bee9ca367ee6468e01b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:01.583726 18908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3510332b7744ca5861cc5bd73d1e408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:01.590929 18992 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: Bootstrap starting.
I20260927 00:43:01.591959 18993 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Bootstrap starting.
I20260927 00:43:01.592594 18991 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Bootstrap starting.
I20260927 00:43:01.593880 18992 tablet_bootstrap.cc:654] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:01.594194 18993 tablet_bootstrap.cc:654] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:01.594703 18991 tablet_bootstrap.cc:654] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:01.594791 18992 log.cc:824] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:01.594879 18993 log.cc:824] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:01.595907 18991 log.cc:824] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:01.597144 18993 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: No bootstrap required, opened a new log
I20260927 00:43:01.597306 18992 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: No bootstrap required, opened a new log
I20260927 00:43:01.597359 18993 ts_tablet_manager.cc:1397] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:43:01.597527 18992 ts_tablet_manager.cc:1397] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 00:43:01.597949 18991 tablet_bootstrap.cc:492] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: No bootstrap required, opened a new log
I20260927 00:43:01.598143 18991 ts_tablet_manager.cc:1397] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:43:01.600520 18993 raft_consensus.cc:348] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.600783 18993 raft_consensus.cc:374] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:01.600855 18993 raft_consensus.cc:729] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351ff89a40c54fbc9fe91b18712fe795, State: Initialized, Role: FOLLOWER
I20260927 00:43:01.601325 18993 consensus_queue.cc:260] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.602007 18840 heartbeater.cc:503] Master 127.0.98.62:44243 was elected leader, sending a full tablet report...
I20260927 00:43:01.602070 18991 raft_consensus.cc:348] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.602038 18992 raft_consensus.cc:348] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.602317 18991 raft_consensus.cc:374] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:01.602317 18992 raft_consensus.cc:374] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:01.602365 18991 raft_consensus.cc:729] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd4a5899c9b4a9cb6801c04b89eadc7, State: Initialized, Role: FOLLOWER
I20260927 00:43:01.602373 18992 raft_consensus.cc:729] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f711e2f5844f709b2ae3b9dff2692a, State: Initialized, Role: FOLLOWER
I20260927 00:43:01.602417 18993 ts_tablet_manager.cc:1428] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:43:01.602898 18992 consensus_queue.cc:260] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.603631 18706 heartbeater.cc:503] Master 127.0.98.62:44243 was elected leader, sending a full tablet report...
I20260927 00:43:01.603544 18991 consensus_queue.cc:260] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.604662 18991 ts_tablet_manager.cc:1428] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 00:43:01.604936 18992 ts_tablet_manager.cc:1428] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:43:01.605054 18974 heartbeater.cc:503] Master 127.0.98.62:44243 was elected leader, sending a full tablet report...
I20260927 00:43:01.612605 18997 raft_consensus.cc:482] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:01.612828 18997 raft_consensus.cc:504] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.613904 18997 leader_election.cc:288] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f711e2f5844f709b2ae3b9dff2692a (127.0.98.1:43471), dfd4a5899c9b4a9cb6801c04b89eadc7 (127.0.98.3:41281)
I20260927 00:43:01.617641 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.614507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59959 (local address 127.0.98.3:41281)
0927 00:43:01.614621 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.614625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.614635 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:43:01.614676 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.614678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.614718 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.614780 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:01.615054 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.615577 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.616674 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.617109 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.617201 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.617266 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.617443 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.617516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:43:01.617660 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.614129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41281 (local address 127.0.98.2:59959)
0927 00:43:01.614321 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.614336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.614424 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.614812 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.614815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.614832 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:01.614940 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.614945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.615683 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.615686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.616558 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.616568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.617219 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.617223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.617325 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.617346 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.617407 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.617497 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 00:43:01.617681 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.614233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39969 (local address 127.0.98.1:43471)
0927 00:43:01.614353 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.614357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.614622 (+ 265us) server_negotiation.cc:407] Connection header received
0927 00:43:01.614754 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.614756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.614800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.614859 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:01.615126 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.615621 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.616705 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.617163 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.617258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.617323 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.617491 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.617566 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:43:01.617697 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.614129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.98.2:39969)
0927 00:43:01.614526 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.614536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:01.614654 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:01.614875 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:01.614878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:01.614894 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:01.615023 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.615028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.615732 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.615736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.616590 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:01.616599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.617273 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.617276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:01.617375 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.617394 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:43:01.617456 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:43:01.617527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":31,"thread_start_us":129,"threads_started":1}
I20260927 00:43:01.618322 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3510332b7744ca5861cc5bd73d1e408" candidate_uuid: "351ff89a40c54fbc9fe91b18712fe795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" is_pre_election: true
I20260927 00:43:01.618348 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3510332b7744ca5861cc5bd73d1e408" candidate_uuid: "351ff89a40c54fbc9fe91b18712fe795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f711e2f5844f709b2ae3b9dff2692a" is_pre_election: true
I20260927 00:43:01.618619 18928 raft_consensus.cc:2454] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351ff89a40c54fbc9fe91b18712fe795 in term 0.
I20260927 00:43:01.618623 18660 raft_consensus.cc:2454] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351ff89a40c54fbc9fe91b18712fe795 in term 0.
I20260927 00:43:01.619081 18727 leader_election.cc:302] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351ff89a40c54fbc9fe91b18712fe795, 64f711e2f5844f709b2ae3b9dff2692a; no voters:
I20260927 00:43:01.619349 18997 raft_consensus.cc:2781] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:01.619426 18997 raft_consensus.cc:482] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:01.619477 18997 raft_consensus.cc:3037] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:01.620615 18997 raft_consensus.cc:504] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.620972 18997 leader_election.cc:288] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [CANDIDATE]: Term 1 election: Requested vote from peers 64f711e2f5844f709b2ae3b9dff2692a (127.0.98.1:43471), dfd4a5899c9b4a9cb6801c04b89eadc7 (127.0.98.3:41281)
I20260927 00:43:01.621325 18928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3510332b7744ca5861cc5bd73d1e408" candidate_uuid: "351ff89a40c54fbc9fe91b18712fe795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7"
I20260927 00:43:01.621342 18660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3510332b7744ca5861cc5bd73d1e408" candidate_uuid: "351ff89a40c54fbc9fe91b18712fe795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f711e2f5844f709b2ae3b9dff2692a"
I20260927 00:43:01.621445 18928 raft_consensus.cc:3037] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:01.621481 18660 raft_consensus.cc:3037] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:01.622527 18660 raft_consensus.cc:2454] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351ff89a40c54fbc9fe91b18712fe795 in term 1.
I20260927 00:43:01.622546 18928 raft_consensus.cc:2454] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351ff89a40c54fbc9fe91b18712fe795 in term 1.
I20260927 00:43:01.622835 18727 leader_election.cc:302] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351ff89a40c54fbc9fe91b18712fe795, 64f711e2f5844f709b2ae3b9dff2692a; no voters:
I20260927 00:43:01.623045 18997 raft_consensus.cc:2781] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:01.623272 18997 raft_consensus.cc:686] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 1 LEADER]: Becoming Leader. State: Replica: 351ff89a40c54fbc9fe91b18712fe795, State: Running, Role: LEADER
I20260927 00:43:01.623618 18997 consensus_queue.cc:237] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } }
I20260927 00:43:01.627024 18513 catalog_manager.cc:5495] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 reported cstate change: term changed from 0 to 1, leader changed from <none> to 351ff89a40c54fbc9fe91b18712fe795 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "351ff89a40c54fbc9fe91b18712fe795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351ff89a40c54fbc9fe91b18712fe795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 } health_report { overall_health: UNKNOWN } } }
I20260927 00:43:01.644881 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.641060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.0.98.2:45761)
0927 00:43:01.641173 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.641177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.641190 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:01.641281 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.641283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.641336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.641432 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:01.641771 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.642284 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.643178 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.643393 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.643481 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.643808 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:01.644363 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:01.644479 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.644627 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.644665 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:43:01.651440 18928 raft_consensus.cc:1261] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 1 FOLLOWER]: Refusing update from remote peer 351ff89a40c54fbc9fe91b18712fe795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:01.651440 18660 raft_consensus.cc:1261] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 1 FOLLOWER]: Refusing update from remote peer 351ff89a40c54fbc9fe91b18712fe795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:01.652343 18997 consensus_queue.cc:1035] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f711e2f5844f709b2ae3b9dff2692a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:01.652593 19001 consensus_queue.cc:1035] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd4a5899c9b4a9cb6801c04b89eadc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:01.670336 18515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:01.670576 18515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:01.673564 18515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d719209eae7f46d5b33d752ffe07f10a: Sending DeleteTablet for 3 replicas of tablet d3510332b7744ca5861cc5bd73d1e408
I20260927 00:43:01.674582 18774 tablet_service.cc:1514] Processing DeleteTablet for tablet d3510332b7744ca5861cc5bd73d1e408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39474
I20260927 00:43:01.674582 18640 tablet_service.cc:1514] Processing DeleteTablet for tablet d3510332b7744ca5861cc5bd73d1e408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44776
I20260927 00:43:01.675170 19014 tablet_replica.cc:330] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a: stopping tablet replica
I20260927 00:43:01.675170 18908 tablet_service.cc:1514] Processing DeleteTablet for tablet d3510332b7744ca5861cc5bd73d1e408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49324
I20260927 00:43:01.675570 19014 raft_consensus.cc:2229] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:01.675745 19015 tablet_replica.cc:330] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: stopping tablet replica
I20260927 00:43:01.675959 19014 raft_consensus.cc:2258] T d3510332b7744ca5861cc5bd73d1e408 P 64f711e2f5844f709b2ae3b9dff2692a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:01.676162 19015 raft_consensus.cc:2229] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:01.676258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18577
I20260927 00:43:01.676591 19015 raft_consensus.cc:2258] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:01.677012 19013 tablet_replica.cc:330] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: stopping tablet replica
I20260927 00:43:01.677318 19013 raft_consensus.cc:2229] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:01.677706 19013 raft_consensus.cc:2258] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:01.679301 19013 ts_tablet_manager.cc:1905] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:01.679903 19015 ts_tablet_manager.cc:1905] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:01.682201 19013 ts_tablet_manager.cc:1918] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:01.682310 19013 log.cc:1196] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/d3510332b7744ca5861cc5bd73d1e408
I20260927 00:43:01.682638 19013 ts_tablet_manager.cc:1939] T d3510332b7744ca5861cc5bd73d1e408 P 351ff89a40c54fbc9fe91b18712fe795: Deleting consensus metadata
I20260927 00:43:01.682947 19015 ts_tablet_manager.cc:1918] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:01.683106 19015 log.cc:1196] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/d3510332b7744ca5861cc5bd73d1e408
I20260927 00:43:01.683483 19015 ts_tablet_manager.cc:1939] T d3510332b7744ca5861cc5bd73d1e408 P dfd4a5899c9b4a9cb6801c04b89eadc7: Deleting consensus metadata
I20260927 00:43:01.684089 18499 catalog_manager.cc:4841] TS 351ff89a40c54fbc9fe91b18712fe795 (127.0.98.2:45761): tablet d3510332b7744ca5861cc5bd73d1e408 (table test-table [id=19ecc6e054244bee9ca367ee6468e01b]) successfully deleted
W20260927 00:43:01.684464 18499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43471 (error 108)
I20260927 00:43:01.684693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18711
I20260927 00:43:01.686005 18501 catalog_manager.cc:4841] TS dfd4a5899c9b4a9cb6801c04b89eadc7 (127.0.98.3:41281): tablet d3510332b7744ca5861cc5bd73d1e408 (table test-table [id=19ecc6e054244bee9ca367ee6468e01b]) successfully deleted
I20260927 00:43:01.686192 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.685855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43471 (local address 127.0.0.1:44788)
0927 00:43:01.686033 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:01.686101 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 00:43:01.686362 18499 catalog_manager.cc:4584] TS 64f711e2f5844f709b2ae3b9dff2692a (127.0.98.1:43471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3510332b7744ca5861cc5bd73d1e408: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43471: connect: Connection refused (error 111)
I20260927 00:43:01.690361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18845
I20260927 00:43:01.696300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 18483
2026-09-27T00:43:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2201 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:01Z Disabled control of system clock
I20260927 00:43:01.726271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39893
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:43:01.842882 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:01.843180 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:01.843216 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:01.847417 19020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:01.847496 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:01.847510 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:01.847528 19020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:01.847543 19020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:01.852625 19020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:01.853749 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:01.855110 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:01.863593 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:01.863642 19028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:01.863725 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:01.864049 19020 server_base.cc:1034] running on GCE node
I20260927 00:43:01.864758 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:01.866077 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:01.867268 19020 hybrid_clock.cc:648] HybridClock initialized: now 1790469781867244 us; error 45 us; skew 500 ppm
I20260927 00:43:01.869539 19020 webserver.cc:460] Webserver started at http://127.0.98.62:45703/ using document root <none> and password file <none>
I20260927 00:43:01.870234 19020 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:01.870313 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:01.870533 19020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:01.872917 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "a5f97a2efbab405a86b69cff3556888c"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "5e21677ff7ced41dc36d6e603ff6d72c"
server_key_iv: "7ddccb4fb9226eb09bf2c6a65fe4329c"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.873544 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5f97a2efbab405a86b69cff3556888c"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "5e21677ff7ced41dc36d6e603ff6d72c"
server_key_iv: "7ddccb4fb9226eb09bf2c6a65fe4329c"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.877492 19020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:01.880095 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:01.881428 19020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:01.881542 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "a5f97a2efbab405a86b69cff3556888c"
format_stamp: "Formatted at 2026-09-27 00:43:01 on dist-test-slave-xltq"
server_key: "5e21677ff7ced41dc36d6e603ff6d72c"
server_key_iv: "7ddccb4fb9226eb09bf2c6a65fe4329c"
server_key_version: "encryptionkey@0"
I20260927 00:43:01.881666 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:01.897364 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:01.898066 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:01.898241 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:01.905085 19020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39893
I20260927 00:43:01.905115 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39893 every 8 connection(s)
I20260927 00:43:01.906071 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:01.909458 19087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:01.912901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19020
I20260927 00:43:01.913023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:01.913276 392 external_mini_cluster.cc:1428] Setting key 740b4d55dde4fe37e947444a15dcfd06
I20260927 00:43:01.915931 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: Bootstrap starting.
I20260927 00:43:01.918782 19087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:01.919652 19087 log.cc:824] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:01.922081 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: No bootstrap required, opened a new log
I20260927 00:43:01.924278 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:01.914787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.98.62:39893)
0927 00:43:01.915510 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:01.915529 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:01.915572 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:43:01.916457 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:01.916481 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:01.917001 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:01.917429 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:01.917887 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.919309 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.920418 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:01.921088 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:01.921245 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:01.922081 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:01.922087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:01.922105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:01.922611 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:01.922686 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:01.922714 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:01.922813 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:01.922974 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 00:43:01.923270 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260927 00:43:01.926616 19087 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } }
I20260927 00:43:01.926932 19087 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:01.926997 19087 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f97a2efbab405a86b69cff3556888c, State: Initialized, Role: FOLLOWER
I20260927 00:43:01.927409 19087 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } }
I20260927 00:43:01.927552 19087 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:01.927618 19087 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:01.927716 19087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:01.928735 19087 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } }
I20260927 00:43:01.929064 19087 leader_election.cc:302] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f97a2efbab405a86b69cff3556888c; no voters:
I20260927 00:43:01.929363 19087 leader_election.cc:288] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:01.929500 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:01.929723 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [term 1 LEADER]: Becoming Leader. State: Replica: a5f97a2efbab405a86b69cff3556888c, State: Running, Role: LEADER
I20260927 00:43:01.930133 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } }
I20260927 00:43:01.930656 19087 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:01.931721 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f97a2efbab405a86b69cff3556888c. Latest consensus state: current_term: 1 leader_uuid: "a5f97a2efbab405a86b69cff3556888c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } } }
I20260927 00:43:01.931869 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:01.932128 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f97a2efbab405a86b69cff3556888c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f97a2efbab405a86b69cff3556888c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39893 } } }
I20260927 00:43:01.932267 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:01.932356 19101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:01.935446 19101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:01.941579 19101 catalog_manager.cc:1347] Generated new cluster ID: 4b59178e2668483c9d708ab029f867ba
I20260927 00:43:01.941668 19101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:01.952482 19101 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:01.953467 19101 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:01.961032 19101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: Generated new TSK 0
I20260927 00:43:01.961787 19101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:01.966712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--builtin_ntp_servers=127.0.98.20:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:43:02.086293 19111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:02.086629 19111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:02.086694 19111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:02.091037 19111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:02.091120 19111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:02.091198 19111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:02.096460 19111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:02.097656 19111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:02.099136 19111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:02.109615 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.109611 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.109611 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:02.110376 19111 server_base.cc:1034] running on GCE node
I20260927 00:43:02.110951 19111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:02.111712 19111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:02.112906 19111 hybrid_clock.cc:648] HybridClock initialized: now 1790469782112883 us; error 39 us; skew 500 ppm
I20260927 00:43:02.115088 19111 webserver.cc:460] Webserver started at http://127.0.98.1:34005/ using document root <none> and password file <none>
I20260927 00:43:02.115690 19111 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:02.115760 19111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:02.115953 19111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:02.118145 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3d44ee8ad67481490d9cd9a06242f5e"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "5297685f87b125e7c843e31a1ee035a5"
server_key_iv: "3ab48b06f6d1cc25907a26fa194ab756"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.118716 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3d44ee8ad67481490d9cd9a06242f5e"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "5297685f87b125e7c843e31a1ee035a5"
server_key_iv: "3ab48b06f6d1cc25907a26fa194ab756"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.122557 19111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:02.125204 19125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.126478 19111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:02.126653 19111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "b3d44ee8ad67481490d9cd9a06242f5e"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "5297685f87b125e7c843e31a1ee035a5"
server_key_iv: "3ab48b06f6d1cc25907a26fa194ab756"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.126761 19111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:02.167147 19111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:02.167879 19111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:02.168064 19111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:02.168721 19111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:02.169749 19111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:02.169798 19111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.169855 19111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:02.169881 19111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.178750 19111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35393
I20260927 00:43:02.178866 19238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35393 every 8 connection(s)
I20260927 00:43:02.179699 19111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:02.184128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19111
I20260927 00:43:02.184264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:02.184525 392 external_mini_cluster.cc:1428] Setting key 78bd4275ad9b0fcde269c93034ca1f8f
I20260927 00:43:02.186652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--builtin_ntp_servers=127.0.98.20:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:43:02.190536 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.181634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49547 (local address 127.0.98.62:39893)
0927 00:43:02.181805 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.181812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.182994 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 00:43:02.184584 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.184588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.184654 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.184743 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:02.185861 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.186465 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.188237 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.188599 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.188714 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.189976 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:02.189980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:02.189983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:02.189999 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:02.190025 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:02.190042 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:02.190115 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.190299 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.190369 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:43:02.191495 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.182100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39893 (local address 127.0.98.1:49547)
0927 00:43:02.182830 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.182882 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.184238 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.184943 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.184951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.185059 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:02.185665 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.185682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.186636 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.186640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.188055 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.188071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.188787 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.188790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.188962 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.189647 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:02.189649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:02.189696 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:02.189779 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:02.190146 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:02.190155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:02.190161 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.190288 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.190622 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260927 00:43:02.193166 19239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39893
I20260927 00:43:02.193492 19239 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:02.194321 19239 heartbeater.cc:511] Master 127.0.98.62:39893 requested a full tablet report, sending...
I20260927 00:43:02.196328 19051 ts_manager.cc:194] Registered new tserver with Master: b3d44ee8ad67481490d9cd9a06242f5e (127.0.98.1:35393)
I20260927 00:43:02.197624 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49547
W20260927 00:43:02.302824 19242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:02.303114 19242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:02.303155 19242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:02.307337 19242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:02.307402 19242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:02.307556 19242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:02.312662 19242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:02.313800 19242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:02.315135 19242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:02.324808 19247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.324793 19248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.324774 19250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:02.324918 19242 server_base.cc:1034] running on GCE node
I20260927 00:43:02.325481 19242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:02.326112 19242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:02.327283 19242 hybrid_clock.cc:648] HybridClock initialized: now 1790469782327250 us; error 45 us; skew 500 ppm
I20260927 00:43:02.329248 19242 webserver.cc:460] Webserver started at http://127.0.98.2:36525/ using document root <none> and password file <none>
I20260927 00:43:02.329823 19242 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:02.329890 19242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:02.330086 19242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:02.332249 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e2c776a77b4499db6a582f5e1a65b33"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "a380efba88ac1543234e500227ab935c"
server_key_iv: "6b44b3068706397a051b75c898a34b11"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.332803 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e2c776a77b4499db6a582f5e1a65b33"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "a380efba88ac1543234e500227ab935c"
server_key_iv: "6b44b3068706397a051b75c898a34b11"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.336562 19242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 00:43:02.339169 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.340544 19242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:02.340651 19242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "8e2c776a77b4499db6a582f5e1a65b33"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "a380efba88ac1543234e500227ab935c"
server_key_iv: "6b44b3068706397a051b75c898a34b11"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.340762 19242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:02.355504 19242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:02.356284 19242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:02.356477 19242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:02.357125 19242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:02.358100 19242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:02.358145 19242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.358186 19242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:02.358207 19242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.367096 19242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42799
I20260927 00:43:02.367125 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42799 every 8 connection(s)
I20260927 00:43:02.368055 19242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:02.374114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19242
I20260927 00:43:02.374243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:02.374547 392 external_mini_cluster.cc:1428] Setting key 89aac590a2863f6909647a280d81b976
I20260927 00:43:02.376710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--builtin_ntp_servers=127.0.98.20:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:43:02.378266 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.370030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50377 (local address 127.0.98.62:39893)
0927 00:43:02.370179 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.370183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.371303 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 00:43:02.372768 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.372771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.372824 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.372899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:02.373871 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.374663 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.376074 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.376407 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.376517 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.377685 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:02.377689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:02.377692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:02.377707 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:02.377732 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:02.377742 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:02.377825 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.378055 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.378122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:43:02.379223 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.370402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39893 (local address 127.0.98.2:50377)
0927 00:43:02.371131 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.371182 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.372553 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.373099 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.373106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.373198 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:02.373713 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.373726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.374794 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.374798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.375950 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.375963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.376565 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.376568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.376727 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.377376 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:02.377378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:02.377431 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:02.377505 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:02.377861 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:02.377870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:02.377876 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.378020 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.378332 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":115,"threads_started":1}
I20260927 00:43:02.380401 19370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39893
I20260927 00:43:02.380661 19370 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:02.381218 19370 heartbeater.cc:511] Master 127.0.98.62:39893 requested a full tablet report, sending...
I20260927 00:43:02.382329 19051 ts_manager.cc:194] Registered new tserver with Master: 8e2c776a77b4499db6a582f5e1a65b33 (127.0.98.2:42799)
I20260927 00:43:02.383132 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50377
W20260927 00:43:02.490978 19373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:02.491274 19373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:02.491313 19373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:02.495494 19373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:02.495573 19373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:02.495630 19373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:02.500553 19373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:02.501614 19373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:02.502938 19373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:02.513317 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.513317 19381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.513339 19379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:02.513733 19373 server_base.cc:1034] running on GCE node
I20260927 00:43:02.514150 19373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:02.514797 19373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:02.515995 19373 hybrid_clock.cc:648] HybridClock initialized: now 1790469782515966 us; error 43 us; skew 500 ppm
I20260927 00:43:02.518139 19373 webserver.cc:460] Webserver started at http://127.0.98.3:39413/ using document root <none> and password file <none>
I20260927 00:43:02.518759 19373 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:02.518829 19373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:02.519061 19373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:02.521533 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "084d145522a94b1fa55981ec60a6d173"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "92a2d0b795684c5d6a64132df90ca7df"
server_key_iv: "4f26cd52494aa9fd00c79ba67a92ac41"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.522123 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "084d145522a94b1fa55981ec60a6d173"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "92a2d0b795684c5d6a64132df90ca7df"
server_key_iv: "4f26cd52494aa9fd00c79ba67a92ac41"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.526254 19373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 00:43:02.529271 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.530853 19373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:43:02.530992 19373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "084d145522a94b1fa55981ec60a6d173"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-xltq"
server_key: "92a2d0b795684c5d6a64132df90ca7df"
server_key_iv: "4f26cd52494aa9fd00c79ba67a92ac41"
server_key_version: "encryptionkey@0"
I20260927 00:43:02.531138 19373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:02.548624 19373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:02.549407 19373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:02.549614 19373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:02.550256 19373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:02.551496 19373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:02.551554 19373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.551616 19373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:02.551646 19373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.560952 19373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43715
I20260927 00:43:02.561020 19500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43715 every 8 connection(s)
I20260927 00:43:02.561968 19373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:02.563740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19373
I20260927 00:43:02.563832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:02.564126 392 external_mini_cluster.cc:1428] Setting key b888fa9dbf426677404e3907d3268df5
I20260927 00:43:02.572513 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.564081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46119 (local address 127.0.98.62:39893)
0927 00:43:02.564257 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.564261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.565605 (+ 1344us) server_negotiation.cc:407] Connection header received
0927 00:43:02.567080 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.567083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.567133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.567209 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:02.568375 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.568962 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.570225 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.570471 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.570570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.571934 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:02.571937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:02.571939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:02.571955 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:02.571978 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:02.571987 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:02.572067 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.572315 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.572387 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:43:02.573513 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.564566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39893 (local address 127.0.98.3:46119)
0927 00:43:02.565448 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.565506 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.566840 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.567449 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.567457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.567612 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:02.568147 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.568172 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.569098 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.569102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.570087 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.570098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.570652 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.570656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.570821 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.571596 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:02.571601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:02.571651 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:02.571733 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:02.572100 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:02.572110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:02.572116 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.572319 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.572613 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":145,"threads_started":1}
I20260927 00:43:02.574772 19501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39893
I20260927 00:43:02.575026 19501 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:02.575647 19501 heartbeater.cc:511] Master 127.0.98.62:39893 requested a full tablet report, sending...
I20260927 00:43:02.576993 19051 ts_manager.cc:194] Registered new tserver with Master: 084d145522a94b1fa55981ec60a6d173 (127.0.98.3:43715)
I20260927 00:43:02.577648 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46119
I20260927 00:43:02.578496 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:02.584949 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.580687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.0.98.62:39893)
0927 00:43:02.580848 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.580852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.580919 (+ 67us) server_negotiation.cc:407] Connection header received
0927 00:43:02.581028 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.581031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.581070 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.581129 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:02.581574 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.582111 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.583064 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.583281 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.583393 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.584381 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:02.584384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:02.584386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:02.584399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:02.584419 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:02.584427 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:02.584552 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.584733 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.584782 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:43:02.588445 19050 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:02.591614 19050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:02.607136 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.600866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35393 (local address 127.0.0.1:54966)
0927 00:43:02.602135 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.602147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.602222 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.602657 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.602660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.602688 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:02.602842 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.602848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604358 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.604361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.605601 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.605610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.606615 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.606618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.606707 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.606739 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.606837 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.606964 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":52,"spinlock_wait_cycles":3840,"thread_start_us":105,"threads_started":1}
I20260927 00:43:02.607163 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.601042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.0.98.1:35393)
0927 00:43:02.601498 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.601510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.602315 (+ 805us) server_negotiation.cc:407] Connection header received
0927 00:43:02.602381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.602384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.602526 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.602661 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:02.602961 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604187 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.605721 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.606430 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.606552 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.606761 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.606869 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.606979 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":102,"threads_started":1}
I20260927 00:43:02.607424 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.601302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.98.3:43715)
0927 00:43:02.601712 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.601722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.602088 (+ 366us) server_negotiation.cc:407] Connection header received
0927 00:43:02.602324 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.602328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.602502 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.602628 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:02.603004 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604187 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.605989 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.606742 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.606848 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.607030 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.607167 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.607270 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260927 00:43:02.607971 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.600929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43715 (local address 127.0.0.1:36184)
0927 00:43:02.601932 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.601995 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.602158 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.602646 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.602652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.602685 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:02.602843 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.602851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604453 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.604456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.605863 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.605874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.606866 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.606870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.606956 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.606977 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.607760 (+ 783us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.607838 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":200,"spinlock_wait_cycles":2816,"thread_start_us":105,"threads_started":1}
I20260927 00:43:02.608675 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.601697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42799 (local address 127.0.0.1:47488)
0927 00:43:02.601955 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.601996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.602116 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.603276 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.603279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.603295 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:02.603388 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.603394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604376 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.604379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.605875 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.605881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.608291 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.608294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.608392 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.608416 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.608482 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.608552 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":90,"thread_start_us":84,"threads_started":1}
I20260927 00:43:02.609141 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.601842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.0.98.2:42799)
0927 00:43:02.602940 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.602951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.602972 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:43:02.603041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.603045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.603168 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.603283 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:02.603490 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.604253 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.607098 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.607910 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.608022 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.608194 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.608884 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.608982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":103,"threads_started":1}
I20260927 00:43:02.610404 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca9adef87b943038510bad07626e6c2 (DEFAULT_TABLE table=test-table [id=68446cc8ed304e6cb211923ef2c43053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:02.611354 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca9adef87b943038510bad07626e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:02.611146 19435 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca9adef87b943038510bad07626e6c2 (DEFAULT_TABLE table=test-table [id=68446cc8ed304e6cb211923ef2c43053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:02.611593 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca9adef87b943038510bad07626e6c2 (DEFAULT_TABLE table=test-table [id=68446cc8ed304e6cb211923ef2c43053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:02.612155 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca9adef87b943038510bad07626e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:02.612567 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca9adef87b943038510bad07626e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:02.618376 19518 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Bootstrap starting.
I20260927 00:43:02.619014 19519 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Bootstrap starting.
I20260927 00:43:02.620500 19520 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Bootstrap starting.
I20260927 00:43:02.620913 19518 tablet_bootstrap.cc:654] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:02.621011 19519 tablet_bootstrap.cc:654] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:02.621799 19518 log.cc:824] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:02.621799 19519 log.cc:824] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:02.622359 19520 tablet_bootstrap.cc:654] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:02.623042 19520 log.cc:824] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:02.623844 19518 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: No bootstrap required, opened a new log
I20260927 00:43:02.624039 19518 ts_tablet_manager.cc:1397] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:43:02.624881 19520 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: No bootstrap required, opened a new log
I20260927 00:43:02.625061 19520 ts_tablet_manager.cc:1397] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:43:02.625110 19519 tablet_bootstrap.cc:492] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: No bootstrap required, opened a new log
I20260927 00:43:02.625298 19519 ts_tablet_manager.cc:1397] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260927 00:43:02.628240 19520 raft_consensus.cc:348] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.628504 19520 raft_consensus.cc:374] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:02.628553 19520 raft_consensus.cc:729] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084d145522a94b1fa55981ec60a6d173, State: Initialized, Role: FOLLOWER
I20260927 00:43:02.628434 19518 raft_consensus.cc:348] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.628763 19518 raft_consensus.cc:374] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:02.628844 19518 raft_consensus.cc:729] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d44ee8ad67481490d9cd9a06242f5e, State: Initialized, Role: FOLLOWER
I20260927 00:43:02.629072 19520 consensus_queue.cc:260] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.629164 19519 raft_consensus.cc:348] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.629261 19518 consensus_queue.cc:260] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.629392 19519 raft_consensus.cc:374] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:02.629441 19519 raft_consensus.cc:729] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2c776a77b4499db6a582f5e1a65b33, State: Initialized, Role: FOLLOWER
I20260927 00:43:02.629915 19501 heartbeater.cc:503] Master 127.0.98.62:39893 was elected leader, sending a full tablet report...
I20260927 00:43:02.630093 19520 ts_tablet_manager.cc:1428] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:43:02.630229 19518 ts_tablet_manager.cc:1428] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:43:02.630546 19239 heartbeater.cc:503] Master 127.0.98.62:39893 was elected leader, sending a full tablet report...
I20260927 00:43:02.630591 19519 consensus_queue.cc:260] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.631609 19519 ts_tablet_manager.cc:1428] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 00:43:02.632061 19370 heartbeater.cc:503] Master 127.0.98.62:39893 was elected leader, sending a full tablet report...
W20260927 00:43:02.680882 19240 tablet.cc:2369] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:02.812999 19502 tablet.cc:2369] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:02.869203 19371 tablet.cc:2369] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:02.978866 19526 raft_consensus.cc:482] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:02.979127 19526 raft_consensus.cc:504] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.980448 19526 leader_election.cc:288] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3d44ee8ad67481490d9cd9a06242f5e (127.0.98.1:35393), 084d145522a94b1fa55981ec60a6d173 (127.0.98.3:43715)
I20260927 00:43:02.984339 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.980903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46333 (local address 127.0.98.3:43715)
0927 00:43:02.981071 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.981075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.981256 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:43:02.981367 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.981370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.981410 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.981471 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:02.981755 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.982253 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983350 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.983750 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983848 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.983914 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.984113 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.984187 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:02.984364 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.980794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41017 (local address 127.0.98.1:35393)
0927 00:43:02.980910 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.980925 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.981157 (+ 232us) server_negotiation.cc:407] Connection header received
0927 00:43:02.981302 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.981306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.981350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.981489 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:02.981721 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.982222 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983353 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.983775 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983871 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.983932 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.984138 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.984244 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:43:02.984373 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.980708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43715 (local address 127.0.98.2:46333)
0927 00:43:02.981176 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.981187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.981280 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.981488 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.981490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.981514 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:02.981653 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.981658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.982358 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.982362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.983220 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.983231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.983874 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.983980 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.984019 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.984081 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.984166 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":62,"thread_start_us":145,"threads_started":1}
I20260927 00:43:02.984380 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.980708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35393 (local address 127.0.98.2:41017)
0927 00:43:02.981061 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:02.981072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:02.981196 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:02.981437 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:02.981440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:02.981456 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:02.981617 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.981624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.982328 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.982332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.983228 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:02.983238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.983895 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.983899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:02.984028 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.984045 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:43:02.984108 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:43:02.984195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":66,"thread_start_us":105,"threads_started":1}
I20260927 00:43:02.985323 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca9adef87b943038510bad07626e6c2" candidate_uuid: "8e2c776a77b4499db6a582f5e1a65b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d145522a94b1fa55981ec60a6d173" is_pre_election: true
I20260927 00:43:02.985332 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca9adef87b943038510bad07626e6c2" candidate_uuid: "8e2c776a77b4499db6a582f5e1a65b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" is_pre_election: true
I20260927 00:43:02.985641 19193 raft_consensus.cc:2454] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2c776a77b4499db6a582f5e1a65b33 in term 0.
I20260927 00:43:02.985652 19455 raft_consensus.cc:2454] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e2c776a77b4499db6a582f5e1a65b33 in term 0.
I20260927 00:43:02.986176 19258 leader_election.cc:302] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e2c776a77b4499db6a582f5e1a65b33, b3d44ee8ad67481490d9cd9a06242f5e; no voters:
I20260927 00:43:02.986569 19526 raft_consensus.cc:2781] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:02.986667 19526 raft_consensus.cc:482] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:02.986735 19526 raft_consensus.cc:3037] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:02.987772 19526 raft_consensus.cc:504] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.988154 19526 leader_election.cc:288] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [CANDIDATE]: Term 1 election: Requested vote from peers b3d44ee8ad67481490d9cd9a06242f5e (127.0.98.1:35393), 084d145522a94b1fa55981ec60a6d173 (127.0.98.3:43715)
I20260927 00:43:02.988564 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca9adef87b943038510bad07626e6c2" candidate_uuid: "8e2c776a77b4499db6a582f5e1a65b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3d44ee8ad67481490d9cd9a06242f5e"
I20260927 00:43:02.988631 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca9adef87b943038510bad07626e6c2" candidate_uuid: "8e2c776a77b4499db6a582f5e1a65b33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "084d145522a94b1fa55981ec60a6d173"
I20260927 00:43:02.988732 19455 raft_consensus.cc:3037] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:02.988698 19193 raft_consensus.cc:3037] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:02.989753 19455 raft_consensus.cc:2454] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e2c776a77b4499db6a582f5e1a65b33 in term 1.
I20260927 00:43:02.989763 19193 raft_consensus.cc:2454] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e2c776a77b4499db6a582f5e1a65b33 in term 1.
I20260927 00:43:02.990135 19257 leader_election.cc:302] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084d145522a94b1fa55981ec60a6d173, 8e2c776a77b4499db6a582f5e1a65b33; no voters:
I20260927 00:43:02.990329 19526 raft_consensus.cc:2781] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:02.990562 19526 raft_consensus.cc:686] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2c776a77b4499db6a582f5e1a65b33, State: Running, Role: LEADER
I20260927 00:43:02.990931 19526 consensus_queue.cc:237] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } }
I20260927 00:43:02.994877 19050 catalog_manager.cc:5495] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e2c776a77b4499db6a582f5e1a65b33 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8e2c776a77b4499db6a582f5e1a65b33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e2c776a77b4499db6a582f5e1a65b33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42799 } health_report { overall_health: HEALTHY } } }
I20260927 00:43:03.062510 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.058923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.0.98.2:42799)
0927 00:43:03.059055 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:03.059060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:03.059074 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:43:03.059144 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:03.059146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:03.059203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:03.059299 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:03.059575 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.060091 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.060956 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.061171 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.061257 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:03.061577 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:03.062071 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:03.062209 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:03.062333 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 00:43:03.062382 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:03.068645 19193 raft_consensus.cc:1261] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 1 FOLLOWER]: Refusing update from remote peer 8e2c776a77b4499db6a582f5e1a65b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:03.068629 19455 raft_consensus.cc:1261] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 1 FOLLOWER]: Refusing update from remote peer 8e2c776a77b4499db6a582f5e1a65b33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:03.069427 19529 consensus_queue.cc:1035] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3d44ee8ad67481490d9cd9a06242f5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:03.069634 19526 consensus_queue.cc:1035] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "084d145522a94b1fa55981ec60a6d173" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:03.079267 19534 mvcc.cc:205] Tried to move back new op lower bound from 7333764231440842752 to 7333764231134777344. Current Snapshot: MvccSnapshot[applied={T|T < 7333764231440842752}]
I20260927 00:43:03.089854 19051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:03.090098 19051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:03.092758 19051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5f97a2efbab405a86b69cff3556888c: Sending DeleteTablet for 3 replicas of tablet 9ca9adef87b943038510bad07626e6c2
I20260927 00:43:03.093726 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca9adef87b943038510bad07626e6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:03 UTC) from {username='slave'} at 127.0.0.1:54966
I20260927 00:43:03.093750 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca9adef87b943038510bad07626e6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:03 UTC) from {username='slave'} at 127.0.0.1:47488
I20260927 00:43:03.093710 19435 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca9adef87b943038510bad07626e6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:03 UTC) from {username='slave'} at 127.0.0.1:36184
I20260927 00:43:03.094220 19542 tablet_replica.cc:330] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e: stopping tablet replica
I20260927 00:43:03.094504 19543 tablet_replica.cc:330] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: stopping tablet replica
I20260927 00:43:03.094606 19542 raft_consensus.cc:2229] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:03.094820 19543 raft_consensus.cc:2229] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:03.094978 19542 raft_consensus.cc:2258] T 9ca9adef87b943038510bad07626e6c2 P b3d44ee8ad67481490d9cd9a06242f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:03.095247 19543 raft_consensus.cc:2258] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:03.095274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19111
I20260927 00:43:03.096014 19544 tablet_replica.cc:330] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: stopping tablet replica
I20260927 00:43:03.096352 19544 raft_consensus.cc:2229] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:03.096691 19544 raft_consensus.cc:2258] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:03.098670 19544 ts_tablet_manager.cc:1905] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:03.101393 19543 ts_tablet_manager.cc:1905] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:03.101622 19544 ts_tablet_manager.cc:1918] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:03.101745 19544 log.cc:1196] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/9ca9adef87b943038510bad07626e6c2
I20260927 00:43:03.102072 19544 ts_tablet_manager.cc:1939] T 9ca9adef87b943038510bad07626e6c2 P 084d145522a94b1fa55981ec60a6d173: Deleting consensus metadata
I20260927 00:43:03.103539 19035 catalog_manager.cc:4841] TS 084d145522a94b1fa55981ec60a6d173 (127.0.98.3:43715): tablet 9ca9adef87b943038510bad07626e6c2 (table test-table [id=68446cc8ed304e6cb211923ef2c43053]) successfully deleted
W20260927 00:43:03.103922 19036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35393 (error 108)
I20260927 00:43:03.104537 19543 ts_tablet_manager.cc:1918] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:03.104650 19543 log.cc:1196] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/9ca9adef87b943038510bad07626e6c2
I20260927 00:43:03.104908 19543 ts_tablet_manager.cc:1939] T 9ca9adef87b943038510bad07626e6c2 P 8e2c776a77b4499db6a582f5e1a65b33: Deleting consensus metadata
I20260927 00:43:03.105031 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.104701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35393 (local address 127.0.0.1:54970)
0927 00:43:03.104881 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:03.104946 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 00:43:03.105192 19036 catalog_manager.cc:4584] TS b3d44ee8ad67481490d9cd9a06242f5e (127.0.98.1:35393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ca9adef87b943038510bad07626e6c2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35393: connect: Connection refused (error 111)
I20260927 00:43:03.105638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19242
I20260927 00:43:03.111461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19373
I20260927 00:43:03.112653 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.111773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42799 (local address 127.0.0.1:47504)
0927 00:43:03.112486 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:03.112554 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600}
I20260927 00:43:03.116600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19020
2026-09-27T00:43:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1418 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:03 dist-test-slave-xltq krb5kdc[19548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:03 dist-test-slave-xltq krb5kdc[19548](info): set up 2 sockets
Sep 27 00:43:03 dist-test-slave-xltq krb5kdc[19548](info): commencing operation
krb5kdc: starting...
W20260927 00:43:04.084257 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:04.220945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:04.339601 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:04.339936 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:04.339996 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:04.344450 19561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:04.344539 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:04.344566 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:04.344587 19561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:04.344607 19561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:04.350449 19561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:04.351614 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:04.352715 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:04.362279 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.362301 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.362301 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:04.362877 19561 server_base.cc:1034] running on GCE node
I20260927 00:43:04.363538 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:04.364830 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:04.366073 19561 hybrid_clock.cc:648] HybridClock initialized: now 1790469784366056 us; error 47 us; skew 500 ppm
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:04.369228 19561 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:04.370307 19561 webserver.cc:460] Webserver started at http://127.0.98.62:38371/ using document root <none> and password file <none>
I20260927 00:43:04.370944 19561 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:04.371029 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:04.371240 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:04.373425 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "77f56377a22c49fc8d92e6a481a6d84c"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "6a63718516ea3cf7305c974ad0279736"
server_key_iv: "d1b792f6fdd433a390ca4defbd2a9d1f"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.373988 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "77f56377a22c49fc8d92e6a481a6d84c"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "6a63718516ea3cf7305c974ad0279736"
server_key_iv: "d1b792f6fdd433a390ca4defbd2a9d1f"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.378005 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:04.381186 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.382910 19561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:04.383081 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "77f56377a22c49fc8d92e6a481a6d84c"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "6a63718516ea3cf7305c974ad0279736"
server_key_iv: "d1b792f6fdd433a390ca4defbd2a9d1f"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.383208 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:04.400312 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:04.403650 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:04.403841 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:04.412286 19561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36705
I20260927 00:43:04.412385 19628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36705 every 8 connection(s)
I20260927 00:43:04.413414 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:04.417531 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:04.418707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19561
I20260927 00:43:04.418879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:04.419286 392 external_mini_cluster.cc:1428] Setting key 40495baf3cc016dd1a76bd60fa0dbd1c
I20260927 00:43:04.426192 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: Bootstrap starting.
I20260927 00:43:04.429966 19629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: Neither blocks nor log segments found. Creating new log.
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:04.431046 19629 log.cc:824] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:04.434369 19629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: No bootstrap required, opened a new log
I20260927 00:43:04.438479 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.421268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.98.62:36705)
0927 00:43:04.421780 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.421797 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.421839 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:43:04.422675 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.422695 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.423102 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.423455 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:04.424660 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.425850 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.427268 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.428031 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.428303 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.431744 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:04.431752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:04.431774 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:04.431811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:04.434009 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.434803 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.434812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.434820 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.434997 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.435333 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.435336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.435338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.435983 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:04.436236 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.436698 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.437139 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":158,"threads_started":1}
I20260927 00:43:04.439985 19629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } }
I20260927 00:43:04.440402 19629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:04.440451 19629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f56377a22c49fc8d92e6a481a6d84c, State: Initialized, Role: FOLLOWER
I20260927 00:43:04.441064 19629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } }
I20260927 00:43:04.441218 19629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:04.441285 19629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:04.441401 19629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:04.442847 19629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } }
I20260927 00:43:04.443221 19629 leader_election.cc:302] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f56377a22c49fc8d92e6a481a6d84c; no voters:
I20260927 00:43:04.443571 19629 leader_election.cc:288] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:04.443805 19634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:04.444180 19634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [term 1 LEADER]: Becoming Leader. State: Replica: 77f56377a22c49fc8d92e6a481a6d84c, State: Running, Role: LEADER
I20260927 00:43:04.444638 19634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } }
I20260927 00:43:04.445204 19629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:04.446756 19635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77f56377a22c49fc8d92e6a481a6d84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } } }
I20260927 00:43:04.446964 19635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:04.446849 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f56377a22c49fc8d92e6a481a6d84c. Latest consensus state: current_term: 1 leader_uuid: "77f56377a22c49fc8d92e6a481a6d84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f56377a22c49fc8d92e6a481a6d84c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36705 } } }
I20260927 00:43:04.447146 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:04.447481 19643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:04.451526 19643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:04.459378 19643 catalog_manager.cc:1347] Generated new cluster ID: 0683f622f0544de7bc9e185c399a4223
I20260927 00:43:04.459487 19643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:04.466425 19643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:04.467587 19643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:04.480605 19643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: Generated new TSK 0
I20260927 00:43:04.481490 19643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:43:04.515686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:04.638509 19655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:04.638836 19655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:04.638890 19655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:04.643368 19655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:04.643465 19655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:04.643535 19655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:04.649807 19655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:04.650955 19655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:04.652077 19655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:04.663466 19661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.663492 19660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.663504 19663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:04.663985 19655 server_base.cc:1034] running on GCE node
I20260927 00:43:04.664667 19655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:04.665634 19655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:04.666867 19655 hybrid_clock.cc:648] HybridClock initialized: now 1790469784666841 us; error 49 us; skew 500 ppm
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:04.671074 19655 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:04.672727 19655 webserver.cc:460] Webserver started at http://127.0.98.1:46293/ using document root <none> and password file <none>
I20260927 00:43:04.673552 19655 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:04.673641 19655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:04.673920 19655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:04.676865 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ab616e92cb24c8c82d18ded34d6f872"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "64968f8fa675b34531575521b81adf85"
server_key_iv: "ae699896d49661d6ebd2299872999bca"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.677659 19655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ab616e92cb24c8c82d18ded34d6f872"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "64968f8fa675b34531575521b81adf85"
server_key_iv: "ae699896d49661d6ebd2299872999bca"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.683307 19655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:43:04.687310 19670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.689587 19655 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:43:04.689772 19655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "2ab616e92cb24c8c82d18ded34d6f872"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "64968f8fa675b34531575521b81adf85"
server_key_iv: "ae699896d49661d6ebd2299872999bca"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.690018 19655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:04.707840 19655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:04.712949 19655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:04.713207 19655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:04.714059 19655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:04.715370 19655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:04.715442 19655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.715512 19655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:04.715543 19655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.727576 19655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35937
I20260927 00:43:04.727646 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35937 every 8 connection(s)
I20260927 00:43:04.728958 19655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:04.733614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19655
I20260927 00:43:04.733911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:04.734330 392 external_mini_cluster.cc:1428] Setting key 4ebca5a58c5f996f1b7d7f0b9230f5af
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:43:04.747468 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.731616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43875 (local address 127.0.98.62:36705)
0927 00:43:04.731828 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.731832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.733407 (+ 1575us) server_negotiation.cc:407] Connection header received
0927 00:43:04.734670 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.734674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.734738 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.734817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:04.736325 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.736934 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.738293 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.738621 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.738755 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.742390 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:04.742394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:04.742397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:04.742447 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:04.744131 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.745030 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.745034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.745036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.745213 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.745927 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.745932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.745935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.746249 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:04.746364 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.747041 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.747263 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:43:04.748431 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.732136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36705 (local address 127.0.98.1:43875)
0927 00:43:04.733188 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.733259 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.734396 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.735061 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.735070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.735520 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:04.736113 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.736129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.737079 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.737083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.738070 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.738083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.738846 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.738850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.739081 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.739892 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:04.739914 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:04.742074 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:04.744401 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.744409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.744439 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.744817 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.745392 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.745395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.745397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.745689 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.746390 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:04.746397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:04.746692 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.746877 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.747285 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":282,"threads_started":1}
I20260927 00:43:04.750480 19784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36705
I20260927 00:43:04.750849 19784 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:04.751573 19784 heartbeater.cc:511] Master 127.0.98.62:36705 requested a full tablet report, sending...
I20260927 00:43:04.753831 19593 ts_manager.cc:194] Registered new tserver with Master: 2ab616e92cb24c8c82d18ded34d6f872 (127.0.98.1:35937)
I20260927 00:43:04.755329 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:43:04.768280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:04.885404 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:04.885700 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:04.885759 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:04.890080 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:04.890156 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:04.890236 19789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:04.895852 19789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:04.897100 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:04.898146 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:04.908519 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.908571 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.908524 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:04.908896 19789 server_base.cc:1034] running on GCE node
I20260927 00:43:04.909423 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:04.910207 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:04.911408 19789 hybrid_clock.cc:648] HybridClock initialized: now 1790469784911377 us; error 47 us; skew 500 ppm
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:04.914906 19789 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:04.916225 19789 webserver.cc:460] Webserver started at http://127.0.98.2:46433/ using document root <none> and password file <none>
I20260927 00:43:04.917019 19789 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:04.917100 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:04.917374 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:04.920125 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "55819f55d9cd49e1bba02877c5baf5a8"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "01cfaac3cac412fac8d730d57f7322c4"
server_key_iv: "e542dc9c45e15a4937b01262dd1163fb"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.920922 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "55819f55d9cd49e1bba02877c5baf5a8"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "01cfaac3cac412fac8d730d57f7322c4"
server_key_iv: "e542dc9c45e15a4937b01262dd1163fb"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.926581 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:43:04.930339 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.932443 19789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:43:04.932634 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "55819f55d9cd49e1bba02877c5baf5a8"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-xltq"
server_key: "01cfaac3cac412fac8d730d57f7322c4"
server_key_iv: "e542dc9c45e15a4937b01262dd1163fb"
server_key_version: "encryptionkey@0"
I20260927 00:43:04.932812 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:04.949110 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:04.953167 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:04.953413 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:04.954182 19789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:04.955518 19789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:04.955577 19789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.955633 19789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:04.955669 19789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.967229 19789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38833
I20260927 00:43:04.967279 19917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38833 every 8 connection(s)
I20260927 00:43:04.968442 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:04.976400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19789
I20260927 00:43:04.976542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:04.976852 392 external_mini_cluster.cc:1428] Setting key 2be580e9e0ee38d0e2fd1aff555908ee
Sep 27 00:43:04 dist-test-slave-xltq krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:04.985534 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.970686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34001 (local address 127.0.98.62:36705)
0927 00:43:04.970914 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.970919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.972115 (+ 1196us) server_negotiation.cc:407] Connection header received
0927 00:43:04.973313 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.973316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.973388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.973505 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:04.974925 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.975672 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.977389 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.977699 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.977815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.980964 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:04.980968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:04.980970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:04.981005 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:04.982646 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.983435 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.983439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.983440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.983541 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.983975 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.983978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.983980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.984269 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:04.984377 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.985119 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.985322 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:43:04.986749 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.971261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36705 (local address 127.0.98.2:34001)
0927 00:43:04.971882 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.971946 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.973003 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.973697 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.973705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.974145 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:04.974707 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.974721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.975842 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.975845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.977208 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.977221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.977861 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.977864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.978022 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.978596 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:04.978615 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:04.980683 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:04.982860 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.982868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.982891 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.983293 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.983640 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.983644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.983646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.983859 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.984388 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:04.984397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:04.984784 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.985025 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.985540 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":165,"threads_started":1}
I20260927 00:43:04.988664 19918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36705
I20260927 00:43:04.989010 19918 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:43:04.989688 19918 heartbeater.cc:511] Master 127.0.98.62:36705 requested a full tablet report, sending...
I20260927 00:43:04.991477 19593 ts_manager.cc:194] Registered new tserver with Master: 55819f55d9cd49e1bba02877c5baf5a8 (127.0.98.2:38833)
I20260927 00:43:04.992439 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:43:05.008665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:05.124629 19923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:05.124936 19923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:05.124984 19923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:05.129174 19923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:05.129259 19923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:05.129321 19923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:05.134943 19923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:05.136070 19923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:05.137174 19923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:05.147771 19931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:05.147802 19929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:05.147802 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:05.148190 19923 server_base.cc:1034] running on GCE node
I20260927 00:43:05.148782 19923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:05.149549 19923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:05.150754 19923 hybrid_clock.cc:648] HybridClock initialized: now 1790469785150721 us; error 68 us; skew 500 ppm
Sep 27 00:43:05 dist-test-slave-xltq krb5kdc[19548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:05.154237 19923 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:05.155335 19923 webserver.cc:460] Webserver started at http://127.0.98.3:41789/ using document root <none> and password file <none>
I20260927 00:43:05.156034 19923 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:05.156111 19923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:05.156363 19923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:05.158639 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "703c79c203f44b9fb764167cf857adb1"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "1685735c54f5b45892f9235aa1125108"
server_key_iv: "46f7e01b72d847290602bdeb6ff04286"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.159291 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "703c79c203f44b9fb764167cf857adb1"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "1685735c54f5b45892f9235aa1125108"
server_key_iv: "46f7e01b72d847290602bdeb6ff04286"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.163837 19923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 00:43:05.166908 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.168830 19923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:05.168982 19923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "703c79c203f44b9fb764167cf857adb1"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "1685735c54f5b45892f9235aa1125108"
server_key_iv: "46f7e01b72d847290602bdeb6ff04286"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.169094 19923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:05.180945 19923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:05.184073 19923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:05.184278 19923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:05.184891 19923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:05.185781 19923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:05.185834 19923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.185877 19923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:05.185910 19923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.195864 19923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39991
I20260927 00:43:05.195932 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39991 every 8 connection(s)
I20260927 00:43:05.196916 19923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:05.206825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 19923
I20260927 00:43:05.207000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:05.207417 392 external_mini_cluster.cc:1428] Setting key 3caf59767edf9e72b8d309708b387b22
Sep 27 00:43:05 dist-test-slave-xltq krb5kdc[19548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:05.215265 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.199709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59501 (local address 127.0.98.62:36705)
0927 00:43:05.199950 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.199956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.201358 (+ 1402us) server_negotiation.cc:407] Connection header received
0927 00:43:05.202878 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.202883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.202965 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.203085 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:05.204661 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.205304 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.206729 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.206986 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.207137 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.211114 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:05.211117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:05.211120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:05.211151 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:05.213076 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:05.213680 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:05.213683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:05.213684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:05.213758 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:05.214092 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:05.214095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:05.214096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:05.214294 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:05.214378 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.214938 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.215088 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 00:43:05.216158 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.200294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36705 (local address 127.0.98.3:59501)
0927 00:43:05.201124 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.201199 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.202555 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.203306 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.203315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.203796 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:05.204427 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.204446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.205487 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.205491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.206519 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.206532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.207364 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.207373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.207739 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.208662 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:05.208686 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:05.210914 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:05.213222 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:05.213230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:05.213253 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:05.213571 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:05.213840 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:05.213843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:05.213845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:05.213997 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:05.214389 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:05.214397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:05.214692 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.214869 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.215239 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":201,"threads_started":1}
I20260927 00:43:05.217702 20052 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36705
I20260927 00:43:05.218127 20052 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:05.218852 20052 heartbeater.cc:511] Master 127.0.98.62:36705 requested a full tablet report, sending...
I20260927 00:43:05.220526 19593 ts_manager.cc:194] Registered new tserver with Master: 703c79c203f44b9fb764167cf857adb1 (127.0.98.3:39991)
I20260927 00:43:05.221192 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59501
I20260927 00:43:05.222995 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:05.235402 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.225923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.98.62:36705)
0927 00:43:05.226055 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.226058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.226242 (+ 184us) server_negotiation.cc:407] Connection header received
0927 00:43:05.226337 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.226340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.226392 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.226498 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:05.227408 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.228047 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.229172 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.229423 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.229615 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.231492 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:05.231497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:05.231499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:05.231536 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:05.233447 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:05.234057 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:05.234061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:05.234062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:05.234170 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:05.234432 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:05.234434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:05.234435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:05.234696 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:05.234811 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.235148 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.235271 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:05.239413 19593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:05.242918 19593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:05.262378 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.255130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38833 (local address 127.0.0.1:50920)
0927 00:43:05.256262 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.256332 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.256483 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.257367 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.257375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.257402 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:05.257610 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.257618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.258973 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.258977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.260441 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.260452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.261709 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.261861 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.261902 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.262035 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.262195 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":49,"thread_start_us":117,"threads_started":1}
I20260927 00:43:05.262599 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.255902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.98.3:39991)
0927 00:43:05.256294 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.256314 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.256939 (+ 625us) server_negotiation.cc:407] Connection header received
0927 00:43:05.257027 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.257033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.257273 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.257455 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:05.257808 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.259059 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.260713 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.261539 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261682 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.261885 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.262317 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.262439 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
I20260927 00:43:05.262789 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.255269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39991 (local address 127.0.0.1:36722)
0927 00:43:05.256703 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.256719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.256838 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.257454 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.257458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.257479 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:05.257632 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.257639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.259230 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.259234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.260555 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.260566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.261988 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.262095 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.262132 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.262220 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.262659 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"mutex_wait_us":339,"thread_start_us":183,"threads_started":1}
I20260927 00:43:05.263002 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.256534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.0.98.1:35937)
0927 00:43:05.257664 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.257671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.257697 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:43:05.257780 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.257785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.258026 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.258189 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:05.258437 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.259811 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261060 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.261672 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261775 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.261979 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.262493 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.262844 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":160,"threads_started":1}
I20260927 00:43:05.262420 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.256161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35937 (local address 127.0.0.1:33828)
0927 00:43:05.256798 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.256811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.256919 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.258182 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.258186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.258206 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:05.258305 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.258311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.259945 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.259948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.260946 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.260953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.262041 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.262045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.262134 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.262157 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.262220 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.262310 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":1,"thread_start_us":326,"threads_started":1}
I20260927 00:43:05.262470 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.255202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.2:38833)
0927 00:43:05.256839 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.256852 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.256878 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:43:05.256959 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.256966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.257195 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.257328 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:05.257772 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.258834 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.260573 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.261539 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.261675 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.261885 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.262102 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.262253 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":100,"threads_started":1}
I20260927 00:43:05.266484 19986 tablet_service.cc:1467] Processing CreateTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 (DEFAULT_TABLE table=test-table [id=19f7603abae847b3b619f84fa530e617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:05.266479 19718 tablet_service.cc:1467] Processing CreateTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 (DEFAULT_TABLE table=test-table [id=19f7603abae847b3b619f84fa530e617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:05.266795 19852 tablet_service.cc:1467] Processing CreateTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 (DEFAULT_TABLE table=test-table [id=19f7603abae847b3b619f84fa530e617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:05.267786 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a46056565a44e49ba9ecc826bee1a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:05.267786 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a46056565a44e49ba9ecc826bee1a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:05.268194 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a46056565a44e49ba9ecc826bee1a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:05.276819 20070 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Bootstrap starting.
I20260927 00:43:05.277671 20071 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Bootstrap starting.
I20260927 00:43:05.277995 20069 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: Bootstrap starting.
I20260927 00:43:05.280565 20070 tablet_bootstrap.cc:654] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:05.280884 20069 tablet_bootstrap.cc:654] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:05.281002 20071 tablet_bootstrap.cc:654] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:05.281792 20069 log.cc:824] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:05.282181 20071 log.cc:824] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:05.282178 20070 log.cc:824] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:05.284284 20069 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: No bootstrap required, opened a new log
I20260927 00:43:05.284556 20069 ts_tablet_manager.cc:1397] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:43:05.284755 20071 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: No bootstrap required, opened a new log
I20260927 00:43:05.284941 20070 tablet_bootstrap.cc:492] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: No bootstrap required, opened a new log
I20260927 00:43:05.285017 20071 ts_tablet_manager.cc:1397] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:43:05.285172 20070 ts_tablet_manager.cc:1397] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 00:43:05.289350 20071 raft_consensus.cc:348] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.289741 20071 raft_consensus.cc:374] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:05.289804 20071 raft_consensus.cc:729] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55819f55d9cd49e1bba02877c5baf5a8, State: Initialized, Role: FOLLOWER
I20260927 00:43:05.289885 20069 raft_consensus.cc:348] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.290091 20070 raft_consensus.cc:348] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.290287 20069 raft_consensus.cc:374] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:05.290329 20070 raft_consensus.cc:374] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:05.290339 20069 raft_consensus.cc:729] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab616e92cb24c8c82d18ded34d6f872, State: Initialized, Role: FOLLOWER
I20260927 00:43:05.290370 20070 raft_consensus.cc:729] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703c79c203f44b9fb764167cf857adb1, State: Initialized, Role: FOLLOWER
I20260927 00:43:05.290457 20071 consensus_queue.cc:260] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.290947 20069 consensus_queue.cc:260] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.290947 20070 consensus_queue.cc:260] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.291527 19918 heartbeater.cc:503] Master 127.0.98.62:36705 was elected leader, sending a full tablet report...
I20260927 00:43:05.292272 20070 ts_tablet_manager.cc:1428] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:43:05.292272 20069 ts_tablet_manager.cc:1428] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 00:43:05.292368 20071 ts_tablet_manager.cc:1428] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:43:05.292714 19784 heartbeater.cc:503] Master 127.0.98.62:36705 was elected leader, sending a full tablet report...
I20260927 00:43:05.292714 20052 heartbeater.cc:503] Master 127.0.98.62:36705 was elected leader, sending a full tablet report...
I20260927 00:43:05.378577 20076 raft_consensus.cc:482] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:05.378790 20076 raft_consensus.cc:504] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.380023 20076 leader_election.cc:288] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55819f55d9cd49e1bba02877c5baf5a8 (127.0.98.2:38833), 2ab616e92cb24c8c82d18ded34d6f872 (127.0.98.1:35937)
I20260927 00:43:05.384249 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.380365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38833 (local address 127.0.98.3:38375)
0927 00:43:05.380510 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.380526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.380632 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.381100 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.381103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.381121 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:05.381241 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.381248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.382226 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.382230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.383094 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.383104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.383765 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.383872 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.383893 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.383959 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.384044 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:43:05.384254 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.380529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38375 (local address 127.0.98.2:38833)
0927 00:43:05.380665 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.380669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.380727 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:43:05.380788 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.380791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.380837 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.380921 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:05.381373 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.382118 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383227 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.383651 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383745 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.383817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.383988 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.384085 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:43:05.384429 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.380880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42725 (local address 127.0.98.1:35937)
0927 00:43:05.381006 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.381009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.381335 (+ 326us) server_negotiation.cc:407] Connection header received
0927 00:43:05.381501 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.381504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.381552 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.381615 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:05.381853 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.382450 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383451 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.383878 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383981 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.384046 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.384244 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.384324 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:43:05.384442 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.380732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35937 (local address 127.0.98.3:42725)
0927 00:43:05.381259 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.381272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.381396 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.381632 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.381635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.381650 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:05.381754 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.381760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.382553 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.382557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.383326 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.383335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.383989 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.383993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.384096 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.384119 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.384185 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.384288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":129,"threads_started":1}
I20260927 00:43:05.385058 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a46056565a44e49ba9ecc826bee1a10" candidate_uuid: "703c79c203f44b9fb764167cf857adb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab616e92cb24c8c82d18ded34d6f872" is_pre_election: true
I20260927 00:43:05.385059 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a46056565a44e49ba9ecc826bee1a10" candidate_uuid: "703c79c203f44b9fb764167cf857adb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55819f55d9cd49e1bba02877c5baf5a8" is_pre_election: true
I20260927 00:43:05.385412 19872 raft_consensus.cc:2454] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703c79c203f44b9fb764167cf857adb1 in term 0.
I20260927 00:43:05.385412 19738 raft_consensus.cc:2454] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703c79c203f44b9fb764167cf857adb1 in term 0.
I20260927 00:43:05.385914 19942 leader_election.cc:302] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab616e92cb24c8c82d18ded34d6f872, 703c79c203f44b9fb764167cf857adb1; no voters:
I20260927 00:43:05.386210 20076 raft_consensus.cc:2781] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:05.386296 20076 raft_consensus.cc:482] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:05.386327 20076 raft_consensus.cc:3037] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:05.387413 20076 raft_consensus.cc:504] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.387770 20076 leader_election.cc:288] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [CANDIDATE]: Term 1 election: Requested vote from peers 55819f55d9cd49e1bba02877c5baf5a8 (127.0.98.2:38833), 2ab616e92cb24c8c82d18ded34d6f872 (127.0.98.1:35937)
I20260927 00:43:05.388327 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a46056565a44e49ba9ecc826bee1a10" candidate_uuid: "703c79c203f44b9fb764167cf857adb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55819f55d9cd49e1bba02877c5baf5a8"
I20260927 00:43:05.388346 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a46056565a44e49ba9ecc826bee1a10" candidate_uuid: "703c79c203f44b9fb764167cf857adb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab616e92cb24c8c82d18ded34d6f872"
I20260927 00:43:05.388489 19872 raft_consensus.cc:3037] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:05.388579 19738 raft_consensus.cc:3037] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:05.389920 19738 raft_consensus.cc:2454] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703c79c203f44b9fb764167cf857adb1 in term 1.
I20260927 00:43:05.389930 19872 raft_consensus.cc:2454] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703c79c203f44b9fb764167cf857adb1 in term 1.
I20260927 00:43:05.390336 19942 leader_election.cc:302] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab616e92cb24c8c82d18ded34d6f872, 703c79c203f44b9fb764167cf857adb1; no voters:
I20260927 00:43:05.390551 20076 raft_consensus.cc:2781] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:05.390780 20076 raft_consensus.cc:686] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 1 LEADER]: Becoming Leader. State: Replica: 703c79c203f44b9fb764167cf857adb1, State: Running, Role: LEADER
I20260927 00:43:05.391193 20076 consensus_queue.cc:237] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } }
I20260927 00:43:05.394611 19593 catalog_manager.cc:5495] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703c79c203f44b9fb764167cf857adb1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "703c79c203f44b9fb764167cf857adb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703c79c203f44b9fb764167cf857adb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39991 } health_report { overall_health: HEALTHY } } }
I20260927 00:43:05.422223 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.417694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.0.98.3:39991)
0927 00:43:05.417820 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.417825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.417842 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:43:05.417970 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.417974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.418046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.418177 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:05.418534 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.419345 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.420387 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.420743 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.420854 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.421193 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:05.421751 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:05.421886 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.421966 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.422038 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:43:05.429059 19738 raft_consensus.cc:1261] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872 [term 1 FOLLOWER]: Refusing update from remote peer 703c79c203f44b9fb764167cf857adb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:05.429064 19872 raft_consensus.cc:1261] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 1 FOLLOWER]: Refusing update from remote peer 703c79c203f44b9fb764167cf857adb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:05.429899 20076 consensus_queue.cc:1035] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55819f55d9cd49e1bba02877c5baf5a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:05.430146 20079 consensus_queue.cc:1035] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab616e92cb24c8c82d18ded34d6f872" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:05.439306 20083 mvcc.cc:205] Tried to move back new op lower bound from 7333764241108451328 to 7333764240965263360. Current Snapshot: MvccSnapshot[applied={T|T < 7333764241108451328}]
I20260927 00:43:05.451290 19592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:05.451474 19592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:05.453994 19592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77f56377a22c49fc8d92e6a481a6d84c: Sending DeleteTablet for 3 replicas of tablet 8a46056565a44e49ba9ecc826bee1a10
I20260927 00:43:05.454978 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260927 00:43:05.454993 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36722
I20260927 00:43:05.455142 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a46056565a44e49ba9ecc826bee1a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33828
I20260927 00:43:05.455740 20094 tablet_replica.cc:330] T 8a46056565a44e49ba9ecc826bee1a10 P 2ab616e92cb24c8c82d18ded34d6f872: stopping tablet replica
I20260927 00:43:05.455864 20092 tablet_replica.cc:330] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: stopping tablet replica
I20260927 00:43:05.455945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19655
I20260927 00:43:05.456784 20092 raft_consensus.cc:2229] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:05.457324 20092 raft_consensus.cc:2258] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:05.459029 20092 ts_tablet_manager.cc:1905] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:05.460448 20093 tablet_replica.cc:330] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: stopping tablet replica
I20260927 00:43:05.460850 20093 raft_consensus.cc:2229] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:05.461275 20093 raft_consensus.cc:2258] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:05.461957 20092 ts_tablet_manager.cc:1918] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:05.462059 20092 log.cc:1196] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/8a46056565a44e49ba9ecc826bee1a10
W20260927 00:43:05.462260 19580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35937 (error 108)
I20260927 00:43:05.462368 20092 ts_tablet_manager.cc:1939] T 8a46056565a44e49ba9ecc826bee1a10 P 703c79c203f44b9fb764167cf857adb1: Deleting consensus metadata
I20260927 00:43:05.463110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19789
I20260927 00:43:05.463570 20093 ts_tablet_manager.cc:1905] T 8a46056565a44e49ba9ecc826bee1a10 P 55819f55d9cd49e1bba02877c5baf5a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:05.464517 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.464235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35937 (local address 127.0.0.1:33842)
0927 00:43:05.464354 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.464427 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 00:43:05.464699 19580 catalog_manager.cc:4584] TS 2ab616e92cb24c8c82d18ded34d6f872 (127.0.98.1:35937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a46056565a44e49ba9ecc826bee1a10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35937: connect: Connection refused (error 111)
I20260927 00:43:05.466084 19578 catalog_manager.cc:4841] TS 703c79c203f44b9fb764167cf857adb1 (127.0.98.3:39991): tablet 8a46056565a44e49ba9ecc826bee1a10 (table test-table [id=19f7603abae847b3b619f84fa530e617]) successfully deleted
I20260927 00:43:05.470198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19923
I20260927 00:43:05.470429 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.470136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38833 (local address 127.0.0.1:50928)
0927 00:43:05.470281 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.470340 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:43:05.475968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 19561
2026-09-27T00:43:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2361 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:05Z Disabled control of system clock
I20260927 00:43:05.505864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:43:05.620149 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:05.620517 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:05.620558 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:05.624799 20099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:05.624882 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:05.624897 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:05.624913 20099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:05.624927 20099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:05.630031 20099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:05.631134 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:05.632520 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:05.641803 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:05.641868 20099 server_base.cc:1034] running on GCE node
W20260927 00:43:05.641772 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:05.641803 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:05.642627 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:05.643878 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:05.645056 20099 hybrid_clock.cc:648] HybridClock initialized: now 1790469785645026 us; error 62 us; skew 500 ppm
I20260927 00:43:05.647192 20099 webserver.cc:460] Webserver started at http://127.0.98.62:33647/ using document root <none> and password file <none>
I20260927 00:43:05.647802 20099 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:05.647868 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:05.648080 20099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:05.650309 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "b0ae2f125a1546ba802b67982513c407"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "208baee317ed77784e3051a7e7873d34"
server_key_iv: "519ab8efe8111a4a990796be4979847d"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.650905 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0ae2f125a1546ba802b67982513c407"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "208baee317ed77784e3051a7e7873d34"
server_key_iv: "519ab8efe8111a4a990796be4979847d"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.654994 20099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:05.658109 20113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.659720 20099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:43:05.659878 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "b0ae2f125a1546ba802b67982513c407"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "208baee317ed77784e3051a7e7873d34"
server_key_iv: "519ab8efe8111a4a990796be4979847d"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.659997 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:05.672536 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:05.673276 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:05.673481 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:05.680687 20099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43639
I20260927 00:43:05.680752 20165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43639 every 8 connection(s)
I20260927 00:43:05.681718 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:05.682139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20099
I20260927 00:43:05.682260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:05.682557 392 external_mini_cluster.cc:1428] Setting key 0aa184c93dc75d52641a7b8dcdad171e
I20260927 00:43:05.685775 20166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:05.693110 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: Bootstrap starting.
I20260927 00:43:05.694074 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.684161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.62:43639)
0927 00:43:05.684988 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.685012 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.685059 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:43:05.685967 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.685992 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.686511 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.686925 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:05.687385 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.688906 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.689976 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.690682 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.690853 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.691854 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:05.691861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:05.691880 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:05.692459 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:05.692537 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:05.692575 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:05.692667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.692837 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.693126 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":125,"threads_started":1}
I20260927 00:43:05.695834 20166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:05.696780 20166 log.cc:824] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:05.699038 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: No bootstrap required, opened a new log
I20260927 00:43:05.702646 20166 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260927 00:43:05.702909 20166 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:05.702960 20166 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ae2f125a1546ba802b67982513c407, State: Initialized, Role: FOLLOWER
I20260927 00:43:05.703429 20166 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260927 00:43:05.703604 20166 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:05.703675 20166 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:05.703780 20166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:05.704994 20166 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260927 00:43:05.705384 20166 leader_election.cc:302] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ae2f125a1546ba802b67982513c407; no voters:
I20260927 00:43:05.705718 20166 leader_election.cc:288] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:05.705871 20171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:05.706090 20171 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [term 1 LEADER]: Becoming Leader. State: Replica: b0ae2f125a1546ba802b67982513c407, State: Running, Role: LEADER
I20260927 00:43:05.706391 20171 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } }
I20260927 00:43:05.707005 20166 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:05.708107 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ae2f125a1546ba802b67982513c407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260927 00:43:05.708137 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ae2f125a1546ba802b67982513c407. Latest consensus state: current_term: 1 leader_uuid: "b0ae2f125a1546ba802b67982513c407" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ae2f125a1546ba802b67982513c407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43639 } } }
I20260927 00:43:05.708287 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:05.708287 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:05.708695 20180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:05.712384 20180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:05.718626 20180 catalog_manager.cc:1347] Generated new cluster ID: 51a045e410554cdf9667162572dc3ee1
I20260927 00:43:05.718719 20180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:05.743790 20180 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:05.745170 20180 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:05.752645 20180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: Generated new TSK 0
I20260927 00:43:05.753558 20180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:05.760111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:43:05.882333 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:05.882635 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:05.882674 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:05.886945 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:05.887020 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:05.887085 20190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:05.892001 20190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:05.893184 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:05.894539 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:05.904402 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:05.904399 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:05.904402 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:05.904443 20190 server_base.cc:1034] running on GCE node
I20260927 00:43:05.905185 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:05.905840 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:05.907020 20190 hybrid_clock.cc:648] HybridClock initialized: now 1790469785906991 us; error 46 us; skew 500 ppm
I20260927 00:43:05.909111 20190 webserver.cc:460] Webserver started at http://127.0.98.1:32771/ using document root <none> and password file <none>
I20260927 00:43:05.909715 20190 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:05.909775 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:05.909978 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:05.912077 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data/instance:
uuid: "a72ce1624a03460981322c87e73f7321"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "db5df739935a82d208349c5cb7be2c05"
server_key_iv: "1ed4b2f6f91e0a88da9a4349049c3e01"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.912684 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a72ce1624a03460981322c87e73f7321"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "db5df739935a82d208349c5cb7be2c05"
server_key_iv: "1ed4b2f6f91e0a88da9a4349049c3e01"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.916571 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:05.919157 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.920753 20190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:05.920913 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal
uuid: "a72ce1624a03460981322c87e73f7321"
format_stamp: "Formatted at 2026-09-27 00:43:05 on dist-test-slave-xltq"
server_key: "db5df739935a82d208349c5cb7be2c05"
server_key_iv: "1ed4b2f6f91e0a88da9a4349049c3e01"
server_key_version: "encryptionkey@0"
I20260927 00:43:05.921060 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:05.935006 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:05.935737 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:05.935925 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:05.936611 20190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:05.937635 20190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:05.937687 20190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.937741 20190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:05.937772 20190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:05.947117 20190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37697
I20260927 00:43:05.947148 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37697 every 8 connection(s)
I20260927 00:43:05.948141 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:05.957218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20190
I20260927 00:43:05.957336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:05.957624 392 external_mini_cluster.cc:1428] Setting key f177dd13b970a8f8221eb6769d94062f
I20260927 00:43:05.959336 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.950207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45707 (local address 127.0.98.62:43639)
0927 00:43:05.950367 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:05.950372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:05.951723 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 00:43:05.953225 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:05.953228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:05.953287 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:05.953398 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:05.954597 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.955198 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.956698 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.956977 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.957116 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.958724 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:05.958727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:05.958729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:05.958743 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:05.958766 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:05.958775 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:05.958862 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:05.959101 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:43:05.959190 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:43:05.959865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:43:05.960510 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:05.950692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.1:45707)
0927 00:43:05.951520 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:05.951573 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:05.952962 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:05.953676 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:05.953685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:05.953789 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:05.954375 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.954397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.955368 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.955371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.956553 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:05.956566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:05.957186 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:05.957189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:05.957408 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:05.958312 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:05.958314 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:05.958396 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:05.958503 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:05.958877 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:05.958890 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:05.958895 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:43:05.959036 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:43:05.959364 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":138,"threads_started":1}
I20260927 00:43:05.961992 20318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260927 00:43:05.962299 20318 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:05.962910 20318 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260927 00:43:05.964977 20124 ts_manager.cc:194] Registered new tserver with Master: a72ce1624a03460981322c87e73f7321 (127.0.98.1:37697)
I20260927 00:43:05.966931 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45707
W20260927 00:43:06.077613 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:06.077896 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:06.077934 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:06.082186 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:06.082273 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:06.082332 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:06.087481 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:06.088600 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:06.089985 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:06.099951 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:06.099911 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:06.099925 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:06.100603 20321 server_base.cc:1034] running on GCE node
I20260927 00:43:06.101308 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:06.102000 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:06.103189 20321 hybrid_clock.cc:648] HybridClock initialized: now 1790469786103170 us; error 35 us; skew 500 ppm
I20260927 00:43:06.105326 20321 webserver.cc:460] Webserver started at http://127.0.98.2:45191/ using document root <none> and password file <none>
I20260927 00:43:06.105933 20321 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:06.106004 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:06.106212 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:06.108536 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f69f7ba7e0d41028d738c516e003275"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "f482cee0a5d046831dd3d08806b2f05c"
server_key_iv: "d85696127a7329c72fa8c8731698fffe"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.109133 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f69f7ba7e0d41028d738c516e003275"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "f482cee0a5d046831dd3d08806b2f05c"
server_key_iv: "d85696127a7329c72fa8c8731698fffe"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.113190 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:43:06.116088 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.117540 20321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:06.117681 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal
uuid: "2f69f7ba7e0d41028d738c516e003275"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "f482cee0a5d046831dd3d08806b2f05c"
server_key_iv: "d85696127a7329c72fa8c8731698fffe"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.117815 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:06.156471 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:06.157264 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:06.157466 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:06.158093 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:06.159092 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:06.159140 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.159195 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:06.159224 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.168582 20321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40007
I20260927 00:43:06.168640 20448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40007 every 8 connection(s)
I20260927 00:43:06.169569 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:06.179101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20321
I20260927 00:43:06.179230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:06.179497 392 external_mini_cluster.cc:1428] Setting key dea8e4ca8ffa6ca937f9faa22c98da76
I20260927 00:43:06.180534 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.171598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49379 (local address 127.0.98.62:43639)
0927 00:43:06.171777 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.171782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.173314 (+ 1532us) server_negotiation.cc:407] Connection header received
0927 00:43:06.174709 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.174712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.174801 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.174899 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:06.175943 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.176600 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.178014 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.178409 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.178532 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.179883 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:06.179887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:06.179890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:06.179906 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:06.179931 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:06.179942 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:06.180012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.180305 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.180382 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:43:06.181533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--builtin_ntp_servers=127.0.98.20:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:43:06.181537 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.172062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.2:49379)
0927 00:43:06.173126 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.173173 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.174460 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.175096 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.175104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.175200 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:06.175753 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.175767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.176752 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.176755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.177869 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.177883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.178677 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.178684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.178911 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.179573 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:06.179576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:06.179637 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:06.179716 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:06.180053 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:06.180065 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:06.180071 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.180246 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.180530 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":331,"threads_started":1}
I20260927 00:43:06.182843 20449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260927 00:43:06.183161 20449 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:06.183712 20449 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260927 00:43:06.184875 20124 ts_manager.cc:194] Registered new tserver with Master: 2f69f7ba7e0d41028d738c516e003275 (127.0.98.2:40007)
I20260927 00:43:06.185559 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49379
W20260927 00:43:06.295598 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:06.295904 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:06.295943 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:06.300290 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:06.300360 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:06.300419 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:06.305606 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:06.306764 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:06.308558 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:06.318562 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:06.318578 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:06.318558 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:06.318758 20452 server_base.cc:1034] running on GCE node
I20260927 00:43:06.319334 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:06.320058 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:06.321218 20452 hybrid_clock.cc:648] HybridClock initialized: now 1790469786321197 us; error 41 us; skew 500 ppm
I20260927 00:43:06.327293 20452 webserver.cc:460] Webserver started at http://127.0.98.3:41619/ using document root <none> and password file <none>
I20260927 00:43:06.327972 20452 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:06.328050 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:06.328313 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:06.330472 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data/instance:
uuid: "3636f7a647864a83927a26c236baa7df"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "c8a44fa1529481d56d9bcd37f55acf5e"
server_key_iv: "eff6752d6fed5b2f2cb6ae998cdcf30e"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.331009 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3636f7a647864a83927a26c236baa7df"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "c8a44fa1529481d56d9bcd37f55acf5e"
server_key_iv: "eff6752d6fed5b2f2cb6ae998cdcf30e"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.334908 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:06.337572 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.339298 20452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:43:06.339435 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal
uuid: "3636f7a647864a83927a26c236baa7df"
format_stamp: "Formatted at 2026-09-27 00:43:06 on dist-test-slave-xltq"
server_key: "c8a44fa1529481d56d9bcd37f55acf5e"
server_key_iv: "eff6752d6fed5b2f2cb6ae998cdcf30e"
server_key_version: "encryptionkey@0"
I20260927 00:43:06.339571 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:06.355237 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:06.355983 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:06.356166 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:06.356806 20452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:06.357766 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:06.357812 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.357864 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:06.357893 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:06.366739 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33189
I20260927 00:43:06.366760 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33189 every 8 connection(s)
I20260927 00:43:06.367830 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:06.368609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20452
I20260927 00:43:06.368738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:06.369032 392 external_mini_cluster.cc:1428] Setting key e28e658b78beabff47b1e71ddf70e574
I20260927 00:43:06.378003 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.370208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53317 (local address 127.0.98.62:43639)
0927 00:43:06.370335 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.370339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.371680 (+ 1341us) server_negotiation.cc:407] Connection header received
0927 00:43:06.373177 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.373180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.373243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.373316 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:06.374205 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.374733 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.375939 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.376166 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.376287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.377510 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:06.377515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:06.377517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:06.377533 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:06.377558 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:06.377568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:06.377633 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.377778 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.377846 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:43:06.378895 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.370732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43639 (local address 127.0.98.3:53317)
0927 00:43:06.371525 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.371569 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.372967 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.373486 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.373493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.373578 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:06.374068 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.374080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.374852 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.374855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.375819 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.375829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.376339 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.376343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.376518 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.377179 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:43:06.377182 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:06.377240 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:43:06.377319 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:06.377640 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:06.377650 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:06.377656 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.377775 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.378096 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":196,"threads_started":1}
I20260927 00:43:06.380110 20580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43639
I20260927 00:43:06.380369 20580 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:06.380911 20580 heartbeater.cc:511] Master 127.0.98.62:43639 requested a full tablet report, sending...
I20260927 00:43:06.381987 20124 ts_manager.cc:194] Registered new tserver with Master: 3636f7a647864a83927a26c236baa7df (127.0.98.3:33189)
I20260927 00:43:06.382661 20124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53317
I20260927 00:43:06.383551 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:06.389329 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.385588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.98.62:43639)
0927 00:43:06.385727 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.385732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.385839 (+ 107us) server_negotiation.cc:407] Connection header received
0927 00:43:06.385967 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.385970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.386017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.386089 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:06.386355 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.386959 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.387771 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.387993 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.388079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.388906 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:06.388910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:06.388913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:06.388927 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:06.388947 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:06.388956 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:06.389033 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.389123 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.389181 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:43:06.392562 20124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:06.395552 20124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:06.411173 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.404970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.0.98.1:37697)
0927 00:43:06.405773 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.405785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.405811 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:43:06.405891 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.405895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.406147 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.406729 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:06.406881 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408224 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.409719 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.410485 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.410583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.410800 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.410905 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.411017 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":140,"threads_started":1}
I20260927 00:43:06.411172 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.404829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37697 (local address 127.0.0.1:54776)
0927 00:43:06.405282 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.405343 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.405488 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.406330 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.406336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.406362 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:06.406550 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.406559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408306 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.408310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.409580 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.409591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.410605 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.410609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.410718 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.410756 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.410864 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.410998 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":124,"spinlock_wait_cycles":3584,"thread_start_us":112,"threads_started":1}
I20260927 00:43:06.411412 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.404883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.0.1:55044)
0927 00:43:06.405373 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.405383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.405488 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.406742 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.406744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.406761 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:06.406873 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.406879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408356 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.408359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.409832 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.409841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.411022 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.411026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.411108 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.411129 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.411192 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.411260 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":405,"spinlock_wait_cycles":2560,"thread_start_us":141,"threads_started":1}
I20260927 00:43:06.411873 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.405716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.98.3:33189)
0927 00:43:06.406155 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.406167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.406190 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:43:06.406263 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.406268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.406440 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.406597 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:06.406980 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408175 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.409946 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.410671 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.410777 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.410958 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.411606 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.411713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":111,"threads_started":1}
I20260927 00:43:06.411954 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.405908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.0.98.2:40007)
0927 00:43:06.406305 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.406316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.406714 (+ 398us) server_negotiation.cc:407] Connection header received
0927 00:43:06.406808 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.406812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.406991 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.407117 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:06.407288 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408037 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.410208 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.410955 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.411067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.411237 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.411688 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.411776 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":108,"threads_started":1}
I20260927 00:43:06.412233 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.405718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40007 (local address 127.0.0.1:52526)
0927 00:43:06.406525 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.406540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.406619 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.407110 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.407113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.407130 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:06.407204 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.407209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.408154 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.408156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.410084 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.410094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.411291 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.411295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.411376 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.411396 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.411458 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.412082 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":506,"thread_start_us":94,"threads_started":1}
I20260927 00:43:06.414696 20383 tablet_service.cc:1467] Processing CreateTablet for tablet bda1a313687846f4b035949b8a044c39 (DEFAULT_TABLE table=test-table [id=9d6368b3625e48c2817b05f21d2618d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:06.414706 20514 tablet_service.cc:1467] Processing CreateTablet for tablet bda1a313687846f4b035949b8a044c39 (DEFAULT_TABLE table=test-table [id=9d6368b3625e48c2817b05f21d2618d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:06.414706 20252 tablet_service.cc:1467] Processing CreateTablet for tablet bda1a313687846f4b035949b8a044c39 (DEFAULT_TABLE table=test-table [id=9d6368b3625e48c2817b05f21d2618d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:06.415728 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda1a313687846f4b035949b8a044c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:06.415740 20514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda1a313687846f4b035949b8a044c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:06.415740 20252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bda1a313687846f4b035949b8a044c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:06.422739 20597 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Bootstrap starting.
I20260927 00:43:06.423588 20598 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Bootstrap starting.
I20260927 00:43:06.425197 20597 tablet_bootstrap.cc:654] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:06.425697 20598 tablet_bootstrap.cc:654] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:06.426038 20597 log.cc:824] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:06.426350 20598 log.cc:824] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:06.426975 20599 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Bootstrap starting.
I20260927 00:43:06.428450 20597 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: No bootstrap required, opened a new log
I20260927 00:43:06.428663 20597 ts_tablet_manager.cc:1397] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:43:06.429132 20598 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: No bootstrap required, opened a new log
I20260927 00:43:06.429311 20598 ts_tablet_manager.cc:1397] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:43:06.429404 20599 tablet_bootstrap.cc:654] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:06.430253 20599 log.cc:824] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:06.432775 20599 tablet_bootstrap.cc:492] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: No bootstrap required, opened a new log
I20260927 00:43:06.432731 20598 raft_consensus.cc:348] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.433002 20599 ts_tablet_manager.cc:1397] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:43:06.433005 20598 raft_consensus.cc:374] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:06.433068 20598 raft_consensus.cc:729] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72ce1624a03460981322c87e73f7321, State: Initialized, Role: FOLLOWER
I20260927 00:43:06.432890 20597 raft_consensus.cc:348] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.433218 20597 raft_consensus.cc:374] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:06.433377 20597 raft_consensus.cc:729] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f69f7ba7e0d41028d738c516e003275, State: Initialized, Role: FOLLOWER
I20260927 00:43:06.433549 20598 consensus_queue.cc:260] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.433816 20597 consensus_queue.cc:260] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.434433 20318 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
I20260927 00:43:06.434773 20598 ts_tablet_manager.cc:1428] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:43:06.436120 20597 ts_tablet_manager.cc:1428] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:43:06.436549 20449 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
I20260927 00:43:06.437263 20599 raft_consensus.cc:348] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.437507 20599 raft_consensus.cc:374] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:06.437564 20599 raft_consensus.cc:729] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3636f7a647864a83927a26c236baa7df, State: Initialized, Role: FOLLOWER
I20260927 00:43:06.438021 20599 consensus_queue.cc:260] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.438694 20580 heartbeater.cc:503] Master 127.0.98.62:43639 was elected leader, sending a full tablet report...
I20260927 00:43:06.439097 20599 ts_tablet_manager.cc:1428] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 00:43:06.449337 20319 tablet.cc:2369] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:06.528604 20605 raft_consensus.cc:482] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:06.528801 20605 raft_consensus.cc:504] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.529974 20605 leader_election.cc:288] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72ce1624a03460981322c87e73f7321 (127.0.98.1:37697), 2f69f7ba7e0d41028d738c516e003275 (127.0.98.2:40007)
I20260927 00:43:06.534677 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.530628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44129 (local address 127.0.98.2:40007)
0927 00:43:06.530775 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.530780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.531025 (+ 245us) server_negotiation.cc:407] Connection header received
0927 00:43:06.531136 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.531138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.531186 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.531260 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:06.531496 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.532121 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.533502 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.534083 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.534171 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.534220 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.534450 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.534533 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:43:06.534701 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.530384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33541 (local address 127.0.98.1:37697)
0927 00:43:06.530507 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.530511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.530528 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:43:06.530594 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.530597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.530648 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.530739 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:06.531019 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.531837 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.533407 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.534022 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.534149 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.534216 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.534528 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.534602 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:06.534687 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.530232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37697 (local address 127.0.98.3:33541)
0927 00:43:06.530363 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.530378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.530472 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.530753 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.530757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.530775 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:06.530888 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.530895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.531989 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.531992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.533262 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.533275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.534156 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.534160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.534265 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.534340 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.534416 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.534524 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":59}
I20260927 00:43:06.534695 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.530490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40007 (local address 127.0.98.3:44129)
0927 00:43:06.530950 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.530960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:06.531050 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:06.531277 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:06.531279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:06.531291 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:06.531396 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.531400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.532252 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.532256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.533385 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:06.533394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.534204 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.534207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:06.534310 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.534351 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:43:06.534416 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:43:06.534524 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":27,"thread_start_us":124,"threads_started":1}
I20260927 00:43:06.535468 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda1a313687846f4b035949b8a044c39" candidate_uuid: "3636f7a647864a83927a26c236baa7df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f69f7ba7e0d41028d738c516e003275" is_pre_election: true
I20260927 00:43:06.535466 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda1a313687846f4b035949b8a044c39" candidate_uuid: "3636f7a647864a83927a26c236baa7df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72ce1624a03460981322c87e73f7321" is_pre_election: true
I20260927 00:43:06.535830 20272 raft_consensus.cc:2454] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3636f7a647864a83927a26c236baa7df in term 0.
I20260927 00:43:06.535830 20403 raft_consensus.cc:2454] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3636f7a647864a83927a26c236baa7df in term 0.
I20260927 00:43:06.536398 20469 leader_election.cc:302] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f69f7ba7e0d41028d738c516e003275, 3636f7a647864a83927a26c236baa7df; no voters:
I20260927 00:43:06.536667 20605 raft_consensus.cc:2781] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:06.536756 20605 raft_consensus.cc:482] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:06.536787 20605 raft_consensus.cc:3037] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:06.537830 20605 raft_consensus.cc:504] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.538178 20605 leader_election.cc:288] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [CANDIDATE]: Term 1 election: Requested vote from peers a72ce1624a03460981322c87e73f7321 (127.0.98.1:37697), 2f69f7ba7e0d41028d738c516e003275 (127.0.98.2:40007)
I20260927 00:43:06.538620 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda1a313687846f4b035949b8a044c39" candidate_uuid: "3636f7a647864a83927a26c236baa7df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f69f7ba7e0d41028d738c516e003275"
I20260927 00:43:06.538627 20272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bda1a313687846f4b035949b8a044c39" candidate_uuid: "3636f7a647864a83927a26c236baa7df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72ce1624a03460981322c87e73f7321"
I20260927 00:43:06.538759 20272 raft_consensus.cc:3037] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:06.538759 20403 raft_consensus.cc:3037] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:06.540067 20272 raft_consensus.cc:2454] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3636f7a647864a83927a26c236baa7df in term 1.
I20260927 00:43:06.540067 20403 raft_consensus.cc:2454] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3636f7a647864a83927a26c236baa7df in term 1.
I20260927 00:43:06.540442 20469 leader_election.cc:302] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3636f7a647864a83927a26c236baa7df, a72ce1624a03460981322c87e73f7321; no voters:
I20260927 00:43:06.540645 20605 raft_consensus.cc:2781] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:06.540915 20605 raft_consensus.cc:686] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 1 LEADER]: Becoming Leader. State: Replica: 3636f7a647864a83927a26c236baa7df, State: Running, Role: LEADER
I20260927 00:43:06.541343 20605 consensus_queue.cc:237] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } }
I20260927 00:43:06.544520 20124 catalog_manager.cc:5495] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df reported cstate change: term changed from 0 to 1, leader changed from <none> to 3636f7a647864a83927a26c236baa7df (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3636f7a647864a83927a26c236baa7df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3636f7a647864a83927a26c236baa7df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 } health_report { overall_health: UNKNOWN } } }
I20260927 00:43:06.565507 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.561773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.0.98.3:33189)
0927 00:43:06.561912 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:06.561916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:06.561935 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:43:06.562063 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:06.562066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:06.562125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:06.562224 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:06.562461 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.562969 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.563891 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:06.564122 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:06.564271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:06.564600 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:06.565092 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:06.565204 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:06.565273 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:43:06.565334 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:43:06.571939 20403 raft_consensus.cc:1261] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 1 FOLLOWER]: Refusing update from remote peer 3636f7a647864a83927a26c236baa7df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:06.571939 20272 raft_consensus.cc:1261] T bda1a313687846f4b035949b8a044c39 P a72ce1624a03460981322c87e73f7321 [term 1 FOLLOWER]: Refusing update from remote peer 3636f7a647864a83927a26c236baa7df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:06.572760 20605 consensus_queue.cc:1035] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f69f7ba7e0d41028d738c516e003275" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:06.572979 20607 consensus_queue.cc:1035] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72ce1624a03460981322c87e73f7321" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:06.590559 20124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:06.590772 20124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:06.593489 20124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0ae2f125a1546ba802b67982513c407: Sending DeleteTablet for 3 replicas of tablet bda1a313687846f4b035949b8a044c39
I20260927 00:43:06.594341 20514 tablet_service.cc:1514] Processing DeleteTablet for tablet bda1a313687846f4b035949b8a044c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:06 UTC) from {username='slave'} at 127.0.0.1:55044
I20260927 00:43:06.594429 20383 tablet_service.cc:1514] Processing DeleteTablet for tablet bda1a313687846f4b035949b8a044c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:06 UTC) from {username='slave'} at 127.0.0.1:52526
I20260927 00:43:06.594959 20252 tablet_service.cc:1514] Processing DeleteTablet for tablet bda1a313687846f4b035949b8a044c39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:06 UTC) from {username='slave'} at 127.0.0.1:54776
I20260927 00:43:06.595026 20620 tablet_replica.cc:330] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: stopping tablet replica
I20260927 00:43:06.595690 20620 raft_consensus.cc:2229] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:06.596310 20620 raft_consensus.cc:2258] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:06.596436 20621 tablet_replica.cc:330] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: stopping tablet replica
I20260927 00:43:06.596813 20621 raft_consensus.cc:2229] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:06.597255 20621 raft_consensus.cc:2258] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:06.597681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20190
I20260927 00:43:06.598232 20620 ts_tablet_manager.cc:1905] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:06.600805 20621 ts_tablet_manager.cc:1905] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:06.601263 20620 ts_tablet_manager.cc:1918] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:06.601383 20620 log.cc:1196] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-1/wal/wals/bda1a313687846f4b035949b8a044c39
I20260927 00:43:06.601727 20620 ts_tablet_manager.cc:1939] T bda1a313687846f4b035949b8a044c39 P 2f69f7ba7e0d41028d738c516e003275: Deleting consensus metadata
I20260927 00:43:06.603145 20116 catalog_manager.cc:4841] TS 2f69f7ba7e0d41028d738c516e003275 (127.0.98.2:40007): tablet bda1a313687846f4b035949b8a044c39 (table test-table [id=9d6368b3625e48c2817b05f21d2618d1]) successfully deleted
I20260927 00:43:06.604280 20621 ts_tablet_manager.cc:1918] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:06.604398 20621 log.cc:1196] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Deleting WAL directory at /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469528995939-392-0/minicluster-data/ts-2/wal/wals/bda1a313687846f4b035949b8a044c39
I20260927 00:43:06.604704 20621 ts_tablet_manager.cc:1939] T bda1a313687846f4b035949b8a044c39 P 3636f7a647864a83927a26c236baa7df: Deleting consensus metadata
W20260927 00:43:06.605252 20116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37697 (error 108)
I20260927 00:43:06.606082 20114 catalog_manager.cc:4841] TS 3636f7a647864a83927a26c236baa7df (127.0.98.3:33189): tablet bda1a313687846f4b035949b8a044c39 (table test-table [id=9d6368b3625e48c2817b05f21d2618d1]) successfully deleted
I20260927 00:43:06.606138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20321
I20260927 00:43:06.606388 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:06.606073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37697 (local address 127.0.0.1:54780)
0927 00:43:06.606247 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:06.606308 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 00:43:06.607314 20116 catalog_manager.cc:4584] TS a72ce1624a03460981322c87e73f7321 (127.0.98.1:37697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bda1a313687846f4b035949b8a044c39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37697: connect: Connection refused (error 111)
I20260927 00:43:06.611644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20452
I20260927 00:43:06.616545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20099
2026-09-27T00:43:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1138 ms)
[----------] 6 tests from EncryptionPolicyTest (10531 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:06 dist-test-slave-xltq krb5kdc[20626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:06 dist-test-slave-xltq krb5kdc[20626](info): set up 2 sockets
Sep 27 00:43:06 dist-test-slave-xltq krb5kdc[20626](info): commencing operation
krb5kdc: starting...
W20260927 00:43:07.588245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:07 dist-test-slave-xltq krb5kdc[20626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:07.721088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:07.835670 20639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:07.836036 20639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:07.836113 20639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:07.840939 20639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:07.841030 20639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:07.841051 20639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:07.841068 20639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:07.841082 20639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:07.847088 20639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:07.848248 20639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:07.849334 20639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:07.858167 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:07.858093 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:07.858108 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:07.858256 20639 server_base.cc:1034] running on GCE node
I20260927 00:43:07.859001 20639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:07.860270 20639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:07.861476 20639 hybrid_clock.cc:648] HybridClock initialized: now 1790469787861454 us; error 36 us; skew 500 ppm
Sep 27 00:43:07 dist-test-slave-xltq krb5kdc[20626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:07.864462 20639 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:07.865500 20639 webserver.cc:460] Webserver started at http://127.0.0.1:46791/ using document root <none> and password file <none>
I20260927 00:43:07.866058 20639 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:07.866129 20639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:07.866344 20639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:07.868510 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "ee0bff45dc5d491792e386c1c57c9c27"
format_stamp: "Formatted at 2026-09-27 00:43:07 on dist-test-slave-xltq"
server_key: "182ba3136082e2397064eea0839c21b1"
server_key_iv: "873f82569cc7894286d25b2b19af805e"
server_key_version: "encryptionkey@0"
I20260927 00:43:07.869066 20639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee0bff45dc5d491792e386c1c57c9c27"
format_stamp: "Formatted at 2026-09-27 00:43:07 on dist-test-slave-xltq"
server_key: "182ba3136082e2397064eea0839c21b1"
server_key_iv: "873f82569cc7894286d25b2b19af805e"
server_key_version: "encryptionkey@0"
I20260927 00:43:07.872941 20639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260927 00:43:07.875524 20654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:07.876935 20639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:43:07.877070 20639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "ee0bff45dc5d491792e386c1c57c9c27"
format_stamp: "Formatted at 2026-09-27 00:43:07 on dist-test-slave-xltq"
server_key: "182ba3136082e2397064eea0839c21b1"
server_key_iv: "873f82569cc7894286d25b2b19af805e"
server_key_version: "encryptionkey@0"
I20260927 00:43:07.877224 20639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:07.887724 20639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:07.891188 20639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:07.891393 20639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:07.898684 20639 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37381
I20260927 00:43:07.898728 20706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37381 every 8 connection(s)
I20260927 00:43:07.899842 20639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:07.902750 20707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:07.908150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20639
I20260927 00:43:07.908331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:07.908602 392 external_mini_cluster.cc:1428] Setting key 320189394aa8c8135a4ec48aa9b60b9b
I20260927 00:43:07.908771 20707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27: Bootstrap starting.
I20260927 00:43:07.911324 20707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:07.912101 20707 log.cc:824] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:07.914170 20707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27: No bootstrap required, opened a new log
Sep 27 00:43:07 dist-test-slave-xltq krb5kdc[20626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:43:07.918605 20707 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260927 00:43:07.918903 20707 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:07.918960 20707 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0bff45dc5d491792e386c1c57c9c27, State: Initialized, Role: FOLLOWER
I20260927 00:43:07.919411 20707 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260927 00:43:07.919581 20707 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:07.919657 20707 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:07.919759 20707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:07.920823 20707 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260927 00:43:07.921177 20707 leader_election.cc:302] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0bff45dc5d491792e386c1c57c9c27; no voters:
I20260927 00:43:07.921536 20707 leader_election.cc:288] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:07.921787 20712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:07.922004 20712 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [term 1 LEADER]: Becoming Leader. State: Replica: ee0bff45dc5d491792e386c1c57c9c27, State: Running, Role: LEADER
I20260927 00:43:07.922341 20712 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } }
I20260927 00:43:07.922973 20707 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:07.923673 20712 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0bff45dc5d491792e386c1c57c9c27. Latest consensus state: current_term: 1 leader_uuid: "ee0bff45dc5d491792e386c1c57c9c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260927 00:43:07.923791 20712 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:07.924443 20719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:07.924403 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0bff45dc5d491792e386c1c57c9c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bff45dc5d491792e386c1c57c9c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37381 } } }
I20260927 00:43:07.924640 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:07.925046 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:07.910153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52913 (local address 127.0.0.1:37381)
0927 00:43:07.910632 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:07.910648 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:07.910685 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:07.911475 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:07.911498 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:07.911871 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:07.912178 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:07.913059 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:07.914111 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:07.915289 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:07.915799 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:07.915939 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:07.919003 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:07.919014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:07.919034 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:07.919074 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:07.921157 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:07.921813 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:07.921821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:07.921828 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:07.922102 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:07.922469 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:07.922472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:07.922474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:07.923004 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:07.923300 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:07.923642 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 00:43:07.923900 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260927 00:43:07.928355 20719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:07.928620 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:07.935294 20719 catalog_manager.cc:1347] Generated new cluster ID: a089bd2fb66b41b9ac763a5be2cd12cb
I20260927 00:43:07.935461 20719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:07.942942 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:07.930675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52915 (local address 127.0.0.1:37381)
0927 00:43:07.930800 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:07.930804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:07.935623 (+ 4819us) server_negotiation.cc:407] Connection header received
0927 00:43:07.935791 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:07.935793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:07.935837 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:07.935903 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:07.936665 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:07.937296 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:07.938090 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:07.938344 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:07.938432 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:07.939832 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:07.939835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:07.939838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:07.939862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:07.941207 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:07.941677 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:07.941679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:07.941681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:07.941755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:07.942058 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:07.942060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:07.942062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:07.942237 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:07.942369 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:07.942662 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 00:43:07.942769 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260927 00:43:07.951197 20719 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:07.952421 20719 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:07.962144 20719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee0bff45dc5d491792e386c1c57c9c27: Generated new TSK 0
I20260927 00:43:07.963136 20719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:07.972641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20639
W20260927 00:43:07.977478 20624 connection.cc:535] client connection to 127.0.0.1:37381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37381): Connection reset by peer (error 104)
2026-09-27T00:43:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:08 dist-test-slave-xltq krb5kdc[20738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:08 dist-test-slave-xltq krb5kdc[20738](info): set up 2 sockets
Sep 27 00:43:08 dist-test-slave-xltq krb5kdc[20738](info): commencing operation
krb5kdc: starting...
W20260927 00:43:08.948166 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:09 dist-test-slave-xltq krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:09.075902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:09.189790 20751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:09.190081 20751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:09.190140 20751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:09.194271 20751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:09.194345 20751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:09.194373 20751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:09.194394 20751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:09.194415 20751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:09.199949 20751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:09.201077 20751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:09.202095 20751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:09.211118 20757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:09.211089 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:09.211083 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:09.211902 20751 server_base.cc:1034] running on GCE node
I20260927 00:43:09.212604 20751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:09.213907 20751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:09.215087 20751 hybrid_clock.cc:648] HybridClock initialized: now 1790469789215060 us; error 42 us; skew 500 ppm
Sep 27 00:43:09 dist-test-slave-xltq krb5kdc[20738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:09.218088 20751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:09.219087 20751 webserver.cc:460] Webserver started at http://127.0.0.1:36601/ using document root <none> and password file <none>
I20260927 00:43:09.219784 20751 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:09.219854 20751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:09.220048 20751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:09.222270 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "b0b05248c2b6489aadd06431b05dc202"
format_stamp: "Formatted at 2026-09-27 00:43:09 on dist-test-slave-xltq"
server_key: "0e1ba9c7a617fed5a0e097a1fec79595"
server_key_iv: "2481b3868f784b23e2860363198ea9ad"
server_key_version: "encryptionkey@0"
I20260927 00:43:09.222966 20751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0b05248c2b6489aadd06431b05dc202"
format_stamp: "Formatted at 2026-09-27 00:43:09 on dist-test-slave-xltq"
server_key: "0e1ba9c7a617fed5a0e097a1fec79595"
server_key_iv: "2481b3868f784b23e2860363198ea9ad"
server_key_version: "encryptionkey@0"
I20260927 00:43:09.227319 20751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 00:43:09.230255 20766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:09.231590 20751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:09.231736 20751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "b0b05248c2b6489aadd06431b05dc202"
format_stamp: "Formatted at 2026-09-27 00:43:09 on dist-test-slave-xltq"
server_key: "0e1ba9c7a617fed5a0e097a1fec79595"
server_key_iv: "2481b3868f784b23e2860363198ea9ad"
server_key_version: "encryptionkey@0"
I20260927 00:43:09.231855 20751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:09.252601 20751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:09.255793 20751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:09.255981 20751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:09.263273 20751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35183
I20260927 00:43:09.263288 20818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35183 every 8 connection(s)
I20260927 00:43:09.264317 20751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:09.267227 20819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:09.272809 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202: Bootstrap starting.
I20260927 00:43:09.273041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20751
I20260927 00:43:09.273198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:09.273442 392 external_mini_cluster.cc:1428] Setting key 243183ed8c3dd4ff8acabd8bd4edbfbf
I20260927 00:43:09.275358 20819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:09.276091 20819 log.cc:824] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:09.277956 20819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202: No bootstrap required, opened a new log
I20260927 00:43:09.282130 20819 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 00:43:09.282368 20819 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:09.282424 20819 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b05248c2b6489aadd06431b05dc202, State: Initialized, Role: FOLLOWER
I20260927 00:43:09.282612 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:09.274792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36097 (local address 127.0.0.1:35183)
0927 00:43:09.275194 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:09.275210 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:09.275244 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:43:09.275923 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:09.275942 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:09.276347 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:09.276671 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:09.277060 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:09.278042 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:09.279200 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:09.279692 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:09.279812 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:09.280802 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:09.280807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:09.280821 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:09.281215 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:09.281270 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:09.281292 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:09.281393 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:09.281568 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:43:09.281773 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260927 00:43:09.283010 20819 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 00:43:09.283193 20819 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:09.283259 20819 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:09.283351 20819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:09.284379 20819 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 00:43:09.284749 20819 leader_election.cc:302] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b05248c2b6489aadd06431b05dc202; no voters:
I20260927 00:43:09.285079 20819 leader_election.cc:288] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:09.285347 20824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:09.285578 20824 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [term 1 LEADER]: Becoming Leader. State: Replica: b0b05248c2b6489aadd06431b05dc202, State: Running, Role: LEADER
I20260927 00:43:09.285887 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:09.285913 20824 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } }
I20260927 00:43:09.286386 20819 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:09.288527 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b05248c2b6489aadd06431b05dc202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260927 00:43:09.288724 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:09.288815 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b05248c2b6489aadd06431b05dc202. Latest consensus state: current_term: 1 leader_uuid: "b0b05248c2b6489aadd06431b05dc202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b05248c2b6489aadd06431b05dc202" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35183 } } }
I20260927 00:43:09.288906 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:09.289848 20843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:09.293040 20843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:09.296257 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:09.289471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55157 (local address 127.0.0.1:35183)
0927 00:43:09.289606 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:09.289611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:09.292562 (+ 2951us) server_negotiation.cc:407] Connection header received
0927 00:43:09.292714 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:09.292717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:09.292770 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:09.292843 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:09.293282 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:09.293855 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:09.294670 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:09.294901 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:09.294987 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:09.295779 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:09.295782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:09.295784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:09.295797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:09.295822 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:09.295833 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:09.295904 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:09.296044 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 00:43:09.296081 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:09.299610 20843 catalog_manager.cc:1347] Generated new cluster ID: bef20c49d4ba436e8c2948197537840e
I20260927 00:43:09.299691 20843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:09.313054 20843 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:09.314122 20843 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:09.321983 20843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0b05248c2b6489aadd06431b05dc202: Generated new TSK 0
I20260927 00:43:09.322885 20843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:09.335405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20751
W20260927 00:43:09.340230 20736 connection.cc:535] client connection to 127.0.0.1:35183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35183): Connection reset by peer (error 104)
2026-09-27T00:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:09 dist-test-slave-xltq krb5kdc[20850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:09 dist-test-slave-xltq krb5kdc[20850](info): set up 2 sockets
Sep 27 00:43:09 dist-test-slave-xltq krb5kdc[20850](info): commencing operation
krb5kdc: starting...
W20260927 00:43:10.304332 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:10 dist-test-slave-xltq krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:10.427910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:10.542097 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:10.542408 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:10.542469 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:10.547049 20863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:10.547150 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:10.547179 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:10.547200 20863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:10.547221 20863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:10.553432 20863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:10.554610 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:10.555728 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:10.565150 20871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:10.565001 20869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:10.565001 20868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:10.565348 20863 server_base.cc:1034] running on GCE node
I20260927 00:43:10.566072 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:10.567350 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:10.568560 20863 hybrid_clock.cc:648] HybridClock initialized: now 1790469790568562 us; error 77 us; skew 500 ppm
Sep 27 00:43:10 dist-test-slave-xltq krb5kdc[20850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:10.571869 20863 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:10.572986 20863 webserver.cc:460] Webserver started at http://127.0.0.1:37229/ using document root <none> and password file <none>
I20260927 00:43:10.573566 20863 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:10.573642 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:10.573861 20863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:10.576167 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "e3bb19962301455db487a66ca64e53a6"
format_stamp: "Formatted at 2026-09-27 00:43:10 on dist-test-slave-xltq"
server_key: "b2c0417666bfca0003ed4c20afe05f85"
server_key_iv: "c10475fc850e3f2a57c104a52a15f71c"
server_key_version: "encryptionkey@0"
I20260927 00:43:10.576808 20863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3bb19962301455db487a66ca64e53a6"
format_stamp: "Formatted at 2026-09-27 00:43:10 on dist-test-slave-xltq"
server_key: "b2c0417666bfca0003ed4c20afe05f85"
server_key_iv: "c10475fc850e3f2a57c104a52a15f71c"
server_key_version: "encryptionkey@0"
I20260927 00:43:10.581418 20863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:43:10.584471 20878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:10.586040 20863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:43:10.586192 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "e3bb19962301455db487a66ca64e53a6"
format_stamp: "Formatted at 2026-09-27 00:43:10 on dist-test-slave-xltq"
server_key: "b2c0417666bfca0003ed4c20afe05f85"
server_key_iv: "c10475fc850e3f2a57c104a52a15f71c"
server_key_version: "encryptionkey@0"
I20260927 00:43:10.586344 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:10.591229 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:10.594708 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:10.594897 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:10.601778 20863 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36313
I20260927 00:43:10.601780 20930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36313 every 8 connection(s)
I20260927 00:43:10.602810 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:10.604499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20863
I20260927 00:43:10.604650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:10.604940 392 external_mini_cluster.cc:1428] Setting key 98ea6b5c4c95e02a29c7660a85ca75af
I20260927 00:43:10.606149 20931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:10.610618 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:10.606343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33129 (local address 127.0.0.1:36313)
0927 00:43:10.606844 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:10.606860 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:10.606900 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:43:10.607598 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:10.607617 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:10.607925 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:10.608310 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:10.609283 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:10.609288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:10.609301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:10.609313 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:10.609373 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:10.609393 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:10.609539 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:10.609624 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:43:10.609804 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":161,"threads_started":1}
I20260927 00:43:10.612831 20931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6: Bootstrap starting.
I20260927 00:43:10.614084 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:10.615271 20931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:10.616074 20931 log.cc:824] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:10.617931 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:10.616148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36189 (local address 127.0.0.1:36313)
0927 00:43:10.616279 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:10.616283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:10.616568 (+ 285us) server_negotiation.cc:407] Connection header received
0927 00:43:10.616690 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:10.616693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:10.616727 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:10.616813 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:10.617555 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:10.617558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:10.617561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:10.617573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:10.617601 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:10.617614 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:10.617695 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:10.617775 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:43:10.617812 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:43:10.618276 20931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6: No bootstrap required, opened a new log
I20260927 00:43:10.621950 20931 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:43:10.622221 20931 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:10.622298 20931 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bb19962301455db487a66ca64e53a6, State: Initialized, Role: FOLLOWER
I20260927 00:43:10.622710 20931 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:43:10.622839 20931 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:10.622906 20931 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:10.623006 20931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:10.623958 20931 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:43:10.624351 20931 leader_election.cc:302] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3bb19962301455db487a66ca64e53a6; no voters:
I20260927 00:43:10.624722 20931 leader_election.cc:288] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:10.624848 20941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:10.625222 20941 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [term 1 LEADER]: Becoming Leader. State: Replica: e3bb19962301455db487a66ca64e53a6, State: Running, Role: LEADER
I20260927 00:43:10.625548 20941 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:43:10.626019 20931 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:10.627223 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3bb19962301455db487a66ca64e53a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260927 00:43:10.627250 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bb19962301455db487a66ca64e53a6. Latest consensus state: current_term: 1 leader_uuid: "e3bb19962301455db487a66ca64e53a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb19962301455db487a66ca64e53a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260927 00:43:10.627346 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:10.627345 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:10.627827 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:10.630631 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:10.635979 20951 catalog_manager.cc:1347] Generated new cluster ID: e0df9024c17040e6abff8e2da63221ee
I20260927 00:43:10.636114 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:10.660392 20951 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:10.660833 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:10.671134 20951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3bb19962301455db487a66ca64e53a6: Generated new TSK 0
I20260927 00:43:10.671928 20951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:10.681792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20863
2026-09-27T00:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:10 dist-test-slave-xltq krb5kdc[20962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:10 dist-test-slave-xltq krb5kdc[20962](info): set up 2 sockets
Sep 27 00:43:10 dist-test-slave-xltq krb5kdc[20962](info): commencing operation
krb5kdc: starting...
W20260927 00:43:11.648685 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:11 dist-test-slave-xltq krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:11.772168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:11.891891 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:11.892285 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:11.892344 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:11.896735 20975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:11.896826 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:11.896847 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:11.896863 20975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:11.896878 20975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:11.902740 20975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:11.903898 20975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:11.905021 20975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:11.913744 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:11.913908 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:11.913981 20975 server_base.cc:1034] running on GCE node
W20260927 00:43:11.913911 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:11.914738 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:11.915953 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:11.917150 20975 hybrid_clock.cc:648] HybridClock initialized: now 1790469791917119 us; error 38 us; skew 500 ppm
Sep 27 00:43:11 dist-test-slave-xltq krb5kdc[20962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:11.927416 20975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:11.928727 20975 webserver.cc:460] Webserver started at http://127.0.0.1:43713/ using document root <none> and password file <none>
I20260927 00:43:11.929387 20975 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:11.929471 20975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:11.929708 20975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:11.931993 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "382bd2c5b6df4266b0a305bb3df7b049"
format_stamp: "Formatted at 2026-09-27 00:43:11 on dist-test-slave-xltq"
server_key: "c9ed7997883763d045a64718a942d3e2"
server_key_iv: "b9a8b26cf3d636620a27c01c231d6e67"
server_key_version: "encryptionkey@0"
I20260927 00:43:11.932694 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "382bd2c5b6df4266b0a305bb3df7b049"
format_stamp: "Formatted at 2026-09-27 00:43:11 on dist-test-slave-xltq"
server_key: "c9ed7997883763d045a64718a942d3e2"
server_key_iv: "b9a8b26cf3d636620a27c01c231d6e67"
server_key_version: "encryptionkey@0"
I20260927 00:43:11.937100 20975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:11.940009 20990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:11.941627 20975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:43:11.941776 20975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "382bd2c5b6df4266b0a305bb3df7b049"
format_stamp: "Formatted at 2026-09-27 00:43:11 on dist-test-slave-xltq"
server_key: "c9ed7997883763d045a64718a942d3e2"
server_key_iv: "b9a8b26cf3d636620a27c01c231d6e67"
server_key_version: "encryptionkey@0"
I20260927 00:43:11.941898 20975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:11.955650 20975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:11.959257 20975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:11.959453 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:11.966890 20975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41245
I20260927 00:43:11.966914 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41245 every 8 connection(s)
I20260927 00:43:11.967953 20975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:11.970232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 20975
I20260927 00:43:11.970376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:11.970664 392 external_mini_cluster.cc:1428] Setting key e3c753bda21d49fa6f8c6d328368f9c8
I20260927 00:43:11.971297 21043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:11.977679 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049: Bootstrap starting.
Sep 27 00:43:11 dist-test-slave-xltq krb5kdc[20962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:43:11.980387 21043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:11.981205 21043 log.cc:824] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:11.983599 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049: No bootstrap required, opened a new log
I20260927 00:43:11.985737 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:11.972135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49791 (local address 127.0.0.1:41245)
0927 00:43:11.972642 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:11.972658 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:11.972695 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:11.973388 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:11.973405 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:11.973768 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:11.974093 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:11.974922 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:11.975958 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:11.977152 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:11.977714 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:11.977875 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:11.980613 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:11.980619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:11.980634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:11.980677 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:11.982552 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:11.983030 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:11.983035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:11.983040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:11.983212 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:11.983487 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:11.983490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:11.983491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:11.983952 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:11.984147 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:11.984442 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 00:43:11.984761 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":162,"threads_started":1}
I20260927 00:43:11.987470 21043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260927 00:43:11.987825 21043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:11.987880 21043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382bd2c5b6df4266b0a305bb3df7b049, State: Initialized, Role: FOLLOWER
I20260927 00:43:11.988363 21043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260927 00:43:11.988521 21043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:11.988590 21043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:11.988695 21043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:11.988998 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:11.990025 21043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260927 00:43:11.990468 21043 leader_election.cc:302] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382bd2c5b6df4266b0a305bb3df7b049; no voters:
I20260927 00:43:11.990847 21043 leader_election.cc:288] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:11.990970 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:11.991235 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [term 1 LEADER]: Becoming Leader. State: Replica: 382bd2c5b6df4266b0a305bb3df7b049, State: Running, Role: LEADER
I20260927 00:43:11.991640 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260927 00:43:11.992327 21043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:11.993410 21053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382bd2c5b6df4266b0a305bb3df7b049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } } }
I20260927 00:43:11.993553 21053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:11.993822 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382bd2c5b6df4266b0a305bb3df7b049. Latest consensus state: current_term: 1 leader_uuid: "382bd2c5b6df4266b0a305bb3df7b049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382bd2c5b6df4266b0a305bb3df7b049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } } }
I20260927 00:43:11.993909 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:11.994176 21062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:11.998126 21062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:12.001129 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:11.991597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33929 (local address 127.0.0.1:41245)
0927 00:43:11.991704 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:11.991708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:11.991725 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:43:11.991932 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:11.991935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:11.991987 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:11.992073 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:11.993110 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:11.993928 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:11.995127 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:11.995439 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:11.995542 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:11.997248 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:11.997253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:11.997255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:11.997301 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:11.999092 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:11.999598 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:11.999602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:11.999604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:11.999707 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:12.000024 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:12.000028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:12.000030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:12.000336 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:12.000477 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:12.000758 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:43:12.000895 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:43:12.004725 21062 catalog_manager.cc:1347] Generated new cluster ID: 0fb78c159087433bb2b608052f8ec718
I20260927 00:43:12.004908 21062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:12.031894 21062 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:12.032961 21062 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:12.043365 21062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382bd2c5b6df4266b0a305bb3df7b049: Generated new TSK 0
I20260927 00:43:12.044088 21062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:12.048939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 20975
2026-09-27T00:43:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:12 dist-test-slave-xltq krb5kdc[21074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:12 dist-test-slave-xltq krb5kdc[21074](info): set up 2 sockets
Sep 27 00:43:12 dist-test-slave-xltq krb5kdc[21074](info): commencing operation
krb5kdc: starting...
W20260927 00:43:13.016249 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:13 dist-test-slave-xltq krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:13.139410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:13.253371 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:13.253688 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:13.253751 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:13.257953 21087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:13.258045 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:13.258073 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:13.258095 21087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:13.258113 21087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:13.263888 21087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:13.265090 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:13.266179 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:13.275925 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:13.276049 21087 server_base.cc:1034] running on GCE node
W20260927 00:43:13.275997 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:13.275997 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:13.276916 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:13.278124 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:13.279314 21087 hybrid_clock.cc:648] HybridClock initialized: now 1790469793279286 us; error 35 us; skew 500 ppm
Sep 27 00:43:13 dist-test-slave-xltq krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:13.282158 21087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:13.283223 21087 webserver.cc:460] Webserver started at http://127.0.0.1:33577/ using document root <none> and password file <none>
I20260927 00:43:13.283772 21087 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:13.283833 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:13.284003 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:13.286142 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "612a4905f3a54150b46ddecd6014fae0"
format_stamp: "Formatted at 2026-09-27 00:43:13 on dist-test-slave-xltq"
server_key: "c20cb4238b38a4a4564ea293e7cc23cc"
server_key_iv: "cdd14c25ba3a5e4ba5b94d97d3f57803"
server_key_version: "encryptionkey@0"
I20260927 00:43:13.286670 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "612a4905f3a54150b46ddecd6014fae0"
format_stamp: "Formatted at 2026-09-27 00:43:13 on dist-test-slave-xltq"
server_key: "c20cb4238b38a4a4564ea293e7cc23cc"
server_key_iv: "cdd14c25ba3a5e4ba5b94d97d3f57803"
server_key_version: "encryptionkey@0"
I20260927 00:43:13.290441 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:13.293243 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:13.294632 21087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:43:13.294751 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "612a4905f3a54150b46ddecd6014fae0"
format_stamp: "Formatted at 2026-09-27 00:43:13 on dist-test-slave-xltq"
server_key: "c20cb4238b38a4a4564ea293e7cc23cc"
server_key_iv: "cdd14c25ba3a5e4ba5b94d97d3f57803"
server_key_version: "encryptionkey@0"
I20260927 00:43:13.294863 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:13.309108 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:13.312031 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:13.312251 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:13.319103 21087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41885
I20260927 00:43:13.319128 21154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41885 every 8 connection(s)
I20260927 00:43:13.320068 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:13.323137 21155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:13.326212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21087
I20260927 00:43:13.326381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:13.326644 392 external_mini_cluster.cc:1428] Setting key e8269e09a1128e8e7c6488b9cde609e6
I20260927 00:43:13.329634 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: Bootstrap starting.
I20260927 00:43:13.331990 21155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:13.332806 21155 log.cc:824] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:13.334990 21155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: No bootstrap required, opened a new log
I20260927 00:43:13.336042 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:13.328155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36315 (local address 127.0.0.1:41885)
0927 00:43:13.328788 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:13.328804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:13.328842 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:43:13.329478 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:13.329495 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:13.329893 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:13.330246 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:13.330666 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:13.331653 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:13.332626 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:13.333163 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:13.333335 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:13.334124 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:13.334129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:13.334144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:13.334529 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:13.334584 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:13.334696 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:13.334789 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:13.334997 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:43:13.335220 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":169,"threads_started":1}
I20260927 00:43:13.338804 21155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260927 00:43:13.339077 21155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:13.339104 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:13.339133 21155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612a4905f3a54150b46ddecd6014fae0, State: Initialized, Role: FOLLOWER
I20260927 00:43:13.339617 21155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260927 00:43:13.339773 21155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:13.339831 21155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:13.339922 21155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:13.341262 21155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260927 00:43:13.341672 21155 leader_election.cc:302] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 612a4905f3a54150b46ddecd6014fae0; no voters:
I20260927 00:43:13.342044 21155 leader_election.cc:288] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:13.342145 21165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:13.342358 21165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [term 1 LEADER]: Becoming Leader. State: Replica: 612a4905f3a54150b46ddecd6014fae0, State: Running, Role: LEADER
I20260927 00:43:13.342710 21165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } }
I20260927 00:43:13.343559 21155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:13.344380 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 612a4905f3a54150b46ddecd6014fae0. Latest consensus state: current_term: 1 leader_uuid: "612a4905f3a54150b46ddecd6014fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260927 00:43:13.344520 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:13.344794 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "612a4905f3a54150b46ddecd6014fae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612a4905f3a54150b46ddecd6014fae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41885 } } }
I20260927 00:43:13.344884 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:13.346881 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:13.341507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41533 (local address 127.0.0.1:41885)
0927 00:43:13.341652 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:13.341657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:13.341891 (+ 234us) server_negotiation.cc:407] Connection header received
0927 00:43:13.342024 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:13.342027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:13.342082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:13.342169 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:13.342586 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:13.343341 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:13.344576 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:13.344946 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:13.345024 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:13.346463 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:13.346466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:13.346468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:13.346482 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:13.346508 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:13.346519 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:13.346579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:13.346684 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:43:13.346745 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:43:13.348785 21180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:13.348861 21180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:13.348995 21174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:13.349959 21174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:13.356276 21174 catalog_manager.cc:1347] Generated new cluster ID: e4a48d860e3745ee92172d1e9f491e86
I20260927 00:43:13.356381 21174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:13.373981 21174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:13.375008 21174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:13.385377 21174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 612a4905f3a54150b46ddecd6014fae0: Generated new TSK 0
I20260927 00:43:13.386193 21174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:13.397274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21087
2026-09-27T00:43:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:13 dist-test-slave-xltq krb5kdc[21186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:13 dist-test-slave-xltq krb5kdc[21186](info): set up 2 sockets
Sep 27 00:43:13 dist-test-slave-xltq krb5kdc[21186](info): commencing operation
krb5kdc: starting...
W20260927 00:43:14.364454 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:14 dist-test-slave-xltq krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:14.489091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:14.602350 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:14.602664 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:14.602723 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:14.606882 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:14.606971 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:14.606997 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:14.607017 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:14.607038 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:14.612746 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:14.613898 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:14.614938 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:14.624738 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:14.624738 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:14.624728 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:14.624957 21199 server_base.cc:1034] running on GCE node
I20260927 00:43:14.625788 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:14.627032 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:14.628304 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790469794628285 us; error 45 us; skew 500 ppm
Sep 27 00:43:14 dist-test-slave-xltq krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:14.631848 21199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:14.633059 21199 webserver.cc:460] Webserver started at http://127.0.0.1:37131/ using document root <none> and password file <none>
I20260927 00:43:14.633672 21199 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:14.633749 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:14.633956 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:14.636162 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "c4a1576a9e614de49868e779b6b9d487"
format_stamp: "Formatted at 2026-09-27 00:43:14 on dist-test-slave-xltq"
server_key: "943a809873374c3078e444e02306ca89"
server_key_iv: "9bac090ad68cc5bfdb7ec7ee601c1c8b"
server_key_version: "encryptionkey@0"
I20260927 00:43:14.636963 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4a1576a9e614de49868e779b6b9d487"
format_stamp: "Formatted at 2026-09-27 00:43:14 on dist-test-slave-xltq"
server_key: "943a809873374c3078e444e02306ca89"
server_key_iv: "9bac090ad68cc5bfdb7ec7ee601c1c8b"
server_key_version: "encryptionkey@0"
I20260927 00:43:14.641525 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:43:14.644557 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:14.646347 21199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:43:14.646529 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "c4a1576a9e614de49868e779b6b9d487"
format_stamp: "Formatted at 2026-09-27 00:43:14 on dist-test-slave-xltq"
server_key: "943a809873374c3078e444e02306ca89"
server_key_iv: "9bac090ad68cc5bfdb7ec7ee601c1c8b"
server_key_version: "encryptionkey@0"
I20260927 00:43:14.646678 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:14.651638 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:14.655542 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:14.655740 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:14.663554 21199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37469
I20260927 00:43:14.663662 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37469 every 8 connection(s)
I20260927 00:43:14.664705 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:14.666360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21199
I20260927 00:43:14.666551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:14.666944 392 external_mini_cluster.cc:1428] Setting key be10aab2591d661a52ce6eca092ce0a3
I20260927 00:43:14.669319 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:14.673599 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:14.668752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35843 (local address 127.0.0.1:37469)
0927 00:43:14.669272 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:14.669295 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:14.669532 (+ 237us) server_negotiation.cc:407] Connection header received
0927 00:43:14.670346 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:14.670370 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:14.670700 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:14.671044 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:14.672144 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:14.672150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:14.672164 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:14.672177 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:14.672297 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:14.672324 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:14.672491 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:14.672564 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:43:14.672734 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
I20260927 00:43:14.677012 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:14.677551 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: Bootstrap starting.
I20260927 00:43:14.681140 21267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:14.681305 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:14.679282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37137 (local address 127.0.0.1:37469)
0927 00:43:14.679416 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:14.679419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:14.679600 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:43:14.679767 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:14.679769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:14.679808 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:14.679887 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:14.680756 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:14.680758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:14.680761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:14.680772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:14.680802 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:14.680813 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:14.680905 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:14.681062 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:43:14.681108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:43:14.682199 21267 log.cc:824] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:14.685168 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: No bootstrap required, opened a new log
I20260927 00:43:14.689837 21267 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260927 00:43:14.690230 21267 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:14.690286 21267 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a1576a9e614de49868e779b6b9d487, State: Initialized, Role: FOLLOWER
I20260927 00:43:14.690862 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260927 00:43:14.691035 21267 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:14.691113 21267 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:14.691232 21267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:14.692698 21267 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260927 00:43:14.693161 21267 leader_election.cc:302] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4a1576a9e614de49868e779b6b9d487; no voters:
I20260927 00:43:14.693542 21267 leader_election.cc:288] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:14.693746 21277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:14.694065 21277 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [term 1 LEADER]: Becoming Leader. State: Replica: c4a1576a9e614de49868e779b6b9d487, State: Running, Role: LEADER
I20260927 00:43:14.694485 21277 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } }
I20260927 00:43:14.695174 21267 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:14.696549 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4a1576a9e614de49868e779b6b9d487. Latest consensus state: current_term: 1 leader_uuid: "c4a1576a9e614de49868e779b6b9d487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } } }
I20260927 00:43:14.696758 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:14.697005 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4a1576a9e614de49868e779b6b9d487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a1576a9e614de49868e779b6b9d487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37469 } } }
I20260927 00:43:14.697171 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487 [sys.catalog]: This master's current role is: LEADER
W20260927 00:43:14.701540 21292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:14.701692 21292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:14.701808 21286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:14.702772 21286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:14.711275 21286 catalog_manager.cc:1347] Generated new cluster ID: 5f1151baf82b4c708c234375b60e0b90
I20260927 00:43:14.711409 21286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:14.747663 21286 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:14.748158 21286 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:14.756140 21286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4a1576a9e614de49868e779b6b9d487: Generated new TSK 0
I20260927 00:43:14.757263 21286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:14.767627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21199
2026-09-27T00:43:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:14 dist-test-slave-xltq krb5kdc[21298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:14 dist-test-slave-xltq krb5kdc[21298](info): set up 2 sockets
Sep 27 00:43:14 dist-test-slave-xltq krb5kdc[21298](info): commencing operation
krb5kdc: starting...
W20260927 00:43:15.744313 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:15 dist-test-slave-xltq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:15.871040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:15.984998 21311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:15.985402 21311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:15.985630 21311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:15.990108 21311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:15.990197 21311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:15.990226 21311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:15.990247 21311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:15.990268 21311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:15.996603 21311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:15.997795 21311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:15.998868 21311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:16.008183 21316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:16.008174 21319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:16.008364 21311 server_base.cc:1034] running on GCE node
W20260927 00:43:16.008174 21317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:16.009227 21311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:16.010370 21311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:16.011565 21311 hybrid_clock.cc:648] HybridClock initialized: now 1790469796011529 us; error 56 us; skew 500 ppm
Sep 27 00:43:16 dist-test-slave-xltq krb5kdc[21298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:16.014722 21311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:16.015801 21311 webserver.cc:460] Webserver started at http://127.0.0.1:39811/ using document root <none> and password file <none>
I20260927 00:43:16.016436 21311 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:16.016511 21311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:16.016723 21311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:16.018961 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "357c9d08997d48c7a83301a4e6907303"
format_stamp: "Formatted at 2026-09-27 00:43:16 on dist-test-slave-xltq"
server_key: "1de6f65e3a931e3227cfa3cfd88860d4"
server_key_iv: "12ffc5e2583e5eb3e3ee73d4bd1a116f"
server_key_version: "encryptionkey@0"
I20260927 00:43:16.019547 21311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "357c9d08997d48c7a83301a4e6907303"
format_stamp: "Formatted at 2026-09-27 00:43:16 on dist-test-slave-xltq"
server_key: "1de6f65e3a931e3227cfa3cfd88860d4"
server_key_iv: "12ffc5e2583e5eb3e3ee73d4bd1a116f"
server_key_version: "encryptionkey@0"
I20260927 00:43:16.023650 21311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:16.026603 21326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:16.028148 21311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:16.028347 21311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "357c9d08997d48c7a83301a4e6907303"
format_stamp: "Formatted at 2026-09-27 00:43:16 on dist-test-slave-xltq"
server_key: "1de6f65e3a931e3227cfa3cfd88860d4"
server_key_iv: "12ffc5e2583e5eb3e3ee73d4bd1a116f"
server_key_version: "encryptionkey@0"
I20260927 00:43:16.028491 21311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:16.044535 21311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:16.047868 21311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:16.048079 21311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:16.055614 21311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34807
I20260927 00:43:16.055647 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34807 every 8 connection(s)
I20260927 00:43:16.056710 21311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:16.058681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21311
I20260927 00:43:16.058815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:16.059108 392 external_mini_cluster.cc:1428] Setting key 37ccdc7410b934180de589e5f2a24afe
I20260927 00:43:16.060081 21379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:16.066241 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303: Bootstrap starting.
Sep 27 00:43:16 dist-test-slave-xltq krb5kdc[21298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:43:16.069103 21379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:16.069953 21379 log.cc:824] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:16.072041 21379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303: No bootstrap required, opened a new log
I20260927 00:43:16.074007 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:16.060689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:58895 (local address 127.0.0.1:34807)
0927 00:43:16.061237 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:16.061252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:16.061293 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:43:16.062033 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:16.062050 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:16.062415 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:16.062782 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:16.063647 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:16.064741 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:16.065608 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:16.066118 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:16.066273 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:16.069110 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:16.069117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:16.069132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:16.069169 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:16.070967 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:16.071553 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:16.071558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:16.071564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:16.071680 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:16.072034 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:16.072038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:16.072040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:16.072513 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:16.072712 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:16.073015 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 00:43:16.073234 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":162,"threads_started":1}
I20260927 00:43:16.076110 21379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } }
I20260927 00:43:16.076422 21379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:16.076493 21379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357c9d08997d48c7a83301a4e6907303, State: Initialized, Role: FOLLOWER
I20260927 00:43:16.076941 21379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } }
I20260927 00:43:16.077096 21379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:16.077167 21379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:16.077188 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:16.077267 21379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:16.078527 21379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } }
I20260927 00:43:16.078922 21379 leader_election.cc:302] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357c9d08997d48c7a83301a4e6907303; no voters:
I20260927 00:43:16.079277 21379 leader_election.cc:288] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:16.079352 21388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:16.079654 21388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [term 1 LEADER]: Becoming Leader. State: Replica: 357c9d08997d48c7a83301a4e6907303, State: Running, Role: LEADER
I20260927 00:43:16.079973 21388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } }
I20260927 00:43:16.080834 21379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:16.081218 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357c9d08997d48c7a83301a4e6907303. Latest consensus state: current_term: 1 leader_uuid: "357c9d08997d48c7a83301a4e6907303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } } }
I20260927 00:43:16.081332 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:16.081527 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357c9d08997d48c7a83301a4e6907303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357c9d08997d48c7a83301a4e6907303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34807 } } }
I20260927 00:43:16.081614 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:16.081971 21393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:16.085848 21393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:16.088124 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:16.079520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54283 (local address 127.0.0.1:34807)
0927 00:43:16.079632 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:16.079636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:16.079968 (+ 332us) server_negotiation.cc:407] Connection header received
0927 00:43:16.080078 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:16.080081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:16.080127 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:16.080190 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:16.080909 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:16.081647 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:16.082750 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:16.083093 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:16.083191 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:16.084824 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:16.084827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:16.084831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:16.084876 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:16.086371 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:16.086904 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:16.086908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:16.086910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:16.086984 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:16.087309 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:16.087312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:16.087313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:16.087487 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:16.087615 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:16.087890 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 00:43:16.087978 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:43:16.092063 21393 catalog_manager.cc:1347] Generated new cluster ID: 02bad9598ab84795bd93f2b2accfaffc
I20260927 00:43:16.092145 21393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:16.106983 21393 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:16.108121 21393 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:16.116936 21393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 357c9d08997d48c7a83301a4e6907303: Generated new TSK 0
I20260927 00:43:16.117708 21393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:16.126724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21311
W20260927 00:43:16.131299 21296 connection.cc:535] client connection to 127.0.0.1:34807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34807): Connection reset by peer (error 104)
2026-09-27T00:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:16 dist-test-slave-xltq krb5kdc[21410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:16 dist-test-slave-xltq krb5kdc[21410](info): set up 2 sockets
Sep 27 00:43:16 dist-test-slave-xltq krb5kdc[21410](info): commencing operation
krb5kdc: starting...
W20260927 00:43:17.088174 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:17 dist-test-slave-xltq krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:17.211658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:17.323074 21423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:17.323354 21423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:17.323395 21423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:17.327551 21423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:17.327623 21423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:17.327642 21423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:17.327657 21423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:17.327672 21423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:17.333076 21423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:17.334125 21423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:17.335108 21423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:17.343743 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:17.343740 21431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:17.343842 21423 server_base.cc:1034] running on GCE node
W20260927 00:43:17.343747 21428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:17.344679 21423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:17.345917 21423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:17.347114 21423 hybrid_clock.cc:648] HybridClock initialized: now 1790469797347086 us; error 41 us; skew 500 ppm
Sep 27 00:43:17 dist-test-slave-xltq krb5kdc[21410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:17.354455 21423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:17.355518 21423 webserver.cc:460] Webserver started at http://127.0.0.1:36341/ using document root <none> and password file <none>
I20260927 00:43:17.356087 21423 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:17.356164 21423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:17.356400 21423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:17.358496 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "42bda8aafe724bf38c12bdc07e6abfbc"
format_stamp: "Formatted at 2026-09-27 00:43:17 on dist-test-slave-xltq"
server_key: "e791c6d8e3ba983d7a68780e80465ace"
server_key_iv: "dc98296076d403c0bcd80d749634321c"
server_key_version: "encryptionkey@0"
I20260927 00:43:17.359079 21423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "42bda8aafe724bf38c12bdc07e6abfbc"
format_stamp: "Formatted at 2026-09-27 00:43:17 on dist-test-slave-xltq"
server_key: "e791c6d8e3ba983d7a68780e80465ace"
server_key_iv: "dc98296076d403c0bcd80d749634321c"
server_key_version: "encryptionkey@0"
I20260927 00:43:17.362780 21423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:43:17.365332 21438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:17.366614 21423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:43:17.366784 21423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "42bda8aafe724bf38c12bdc07e6abfbc"
format_stamp: "Formatted at 2026-09-27 00:43:17 on dist-test-slave-xltq"
server_key: "e791c6d8e3ba983d7a68780e80465ace"
server_key_iv: "dc98296076d403c0bcd80d749634321c"
server_key_version: "encryptionkey@0"
I20260927 00:43:17.366894 21423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:17.382351 21423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:17.385567 21423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:17.385748 21423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:17.392890 21423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42933
I20260927 00:43:17.392910 21490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42933 every 8 connection(s)
I20260927 00:43:17.393910 21423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:17.396746 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:17.398288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21423
I20260927 00:43:17.398424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:17.398689 392 external_mini_cluster.cc:1428] Setting key cdbbecf2c990b21750425224aa6c70e4
I20260927 00:43:17.403309 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc: Bootstrap starting.
I20260927 00:43:17.406025 21491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:17.406915 21491 log.cc:824] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:17.408895 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:17.400004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:34867 (local address 127.0.0.1:42933)
0927 00:43:17.400456 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:17.400475 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:17.400517 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:43:17.401341 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:17.401365 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:17.401816 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:17.402204 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:17.402616 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:17.404039 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:17.405013 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:17.405674 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:17.405824 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:17.406732 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:17.406738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:17.406755 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:17.407261 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:17.407333 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:17.407363 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:17.407473 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:17.407677 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 00:43:17.407818 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260927 00:43:17.409420 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc: No bootstrap required, opened a new log
I20260927 00:43:17.412276 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:17.413626 21491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260927 00:43:17.413898 21491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:17.413944 21491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bda8aafe724bf38c12bdc07e6abfbc, State: Initialized, Role: FOLLOWER
I20260927 00:43:17.414395 21491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260927 00:43:17.414573 21491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:17.414649 21491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:17.414758 21491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:17.415727 21491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260927 00:43:17.416081 21491 leader_election.cc:302] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42bda8aafe724bf38c12bdc07e6abfbc; no voters:
I20260927 00:43:17.416417 21491 leader_election.cc:288] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:17.416561 21501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:17.416776 21501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [term 1 LEADER]: Becoming Leader. State: Replica: 42bda8aafe724bf38c12bdc07e6abfbc, State: Running, Role: LEADER
I20260927 00:43:17.417111 21501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260927 00:43:17.417762 21491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:17.418938 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42bda8aafe724bf38c12bdc07e6abfbc. Latest consensus state: current_term: 1 leader_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260927 00:43:17.419094 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:17.419344 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:17.414643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49305 (local address 127.0.0.1:42933)
0927 00:43:17.414766 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:17.414771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:17.415022 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:43:17.415155 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:17.415158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:17.415207 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:17.415273 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:17.415546 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:17.416116 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:17.417254 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:17.417601 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:17.417710 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:17.418710 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:17.418714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:17.418717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:17.418731 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:17.418760 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:17.418774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:17.418850 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:17.419175 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 00:43:17.419214 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:17.419548 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bda8aafe724bf38c12bdc07e6abfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260927 00:43:17.419646 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:17.420558 21510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:17.423178 21510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:17.428754 21510 catalog_manager.cc:1347] Generated new cluster ID: 36a9104e1e9746b4a33a2e3dc8756843
I20260927 00:43:17.428844 21510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:17.445626 21510 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:17.447005 21510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:17.457173 21510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42bda8aafe724bf38c12bdc07e6abfbc: Generated new TSK 0
I20260927 00:43:17.457998 21510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:17.470072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21423
W20260927 00:43:17.474906 21408 connection.cc:535] client connection to 127.0.0.1:42933 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42933): Connection reset by peer (error 104)
2026-09-27T00:43:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:17 dist-test-slave-xltq krb5kdc[21522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:17 dist-test-slave-xltq krb5kdc[21522](info): set up 2 sockets
Sep 27 00:43:17 dist-test-slave-xltq krb5kdc[21522](info): commencing operation
krb5kdc: starting...
W20260927 00:43:18.424290 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:18 dist-test-slave-xltq krb5kdc[21522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:18.548837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:18.658564 21535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:18.658877 21535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:18.658921 21535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:18.663102 21535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:18.663188 21535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:18.663208 21535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:18.663223 21535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:18.663237 21535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:18.668812 21535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:18.669873 21535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:18.670855 21535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:18.679566 21543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:18.679594 21541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:18.679596 21540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:18.679996 21535 server_base.cc:1034] running on GCE node
I20260927 00:43:18.680552 21535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:18.681773 21535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:18.683053 21535 hybrid_clock.cc:648] HybridClock initialized: now 1790469798683037 us; error 24 us; skew 500 ppm
Sep 27 00:43:18 dist-test-slave-xltq krb5kdc[21522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:18.690039 21535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:18.691486 21535 webserver.cc:460] Webserver started at http://127.0.0.1:41089/ using document root <none> and password file <none>
I20260927 00:43:18.692318 21535 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:18.692422 21535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:18.692679 21535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:18.695650 21535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "8dc72401573b4ed09d731867d5cedfc6"
format_stamp: "Formatted at 2026-09-27 00:43:18 on dist-test-slave-xltq"
server_key: "1b56f9e9097fada1d0dca76c778bfdf3"
server_key_iv: "f6c78548a24893cdb192396cd584cc94"
server_key_version: "encryptionkey@0"
I20260927 00:43:18.696475 21535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "8dc72401573b4ed09d731867d5cedfc6"
format_stamp: "Formatted at 2026-09-27 00:43:18 on dist-test-slave-xltq"
server_key: "1b56f9e9097fada1d0dca76c778bfdf3"
server_key_iv: "f6c78548a24893cdb192396cd584cc94"
server_key_version: "encryptionkey@0"
I20260927 00:43:18.701633 21535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:43:18.704815 21550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:18.706631 21535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:43:18.706791 21535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "8dc72401573b4ed09d731867d5cedfc6"
format_stamp: "Formatted at 2026-09-27 00:43:18 on dist-test-slave-xltq"
server_key: "1b56f9e9097fada1d0dca76c778bfdf3"
server_key_iv: "f6c78548a24893cdb192396cd584cc94"
server_key_version: "encryptionkey@0"
I20260927 00:43:18.706954 21535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:18.713752 21535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:18.717772 21535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:18.717986 21535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:18.726305 21535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44383
I20260927 00:43:18.726732 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44383 every 8 connection(s)
I20260927 00:43:18.727494 21535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:18.731431 21603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:18.735638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21535
I20260927 00:43:18.735816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:18.736101 392 external_mini_cluster.cc:1428] Setting key 317cd3c32355878bfaf68d465da1d7d9
I20260927 00:43:18.739156 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6: Bootstrap starting.
I20260927 00:43:18.741991 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:18.742583 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:18.737657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:52103 (local address 127.0.0.1:44383)
0927 00:43:18.738208 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:18.738227 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:18.738385 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:43:18.739321 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:18.739344 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:18.739712 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:18.740157 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:18.741028 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:18.741034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:18.741048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:18.741060 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:18.741126 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:18.741151 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:18.741281 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:18.741476 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:43:18.741636 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":172,"threads_started":1}
I20260927 00:43:18.742872 21603 log.cc:824] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:18.745185 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6: No bootstrap required, opened a new log
I20260927 00:43:18.745643 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:18.749208 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:18.747615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:32897 (local address 127.0.0.1:44383)
0927 00:43:18.747724 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:18.747729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:18.747886 (+ 157us) server_negotiation.cc:407] Connection header received
0927 00:43:18.748006 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:18.748009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:18.748043 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:18.748107 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:18.748833 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:18.748836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:18.748839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:18.748852 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:18.748879 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:18.748890 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:18.748975 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:18.749046 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:43:18.749089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:43:18.749187 21603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260927 00:43:18.749518 21603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:18.749576 21603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc72401573b4ed09d731867d5cedfc6, State: Initialized, Role: FOLLOWER
I20260927 00:43:18.750051 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260927 00:43:18.750241 21603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:18.750334 21603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:18.750437 21603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:18.751354 21603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260927 00:43:18.751680 21603 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc72401573b4ed09d731867d5cedfc6; no voters:
I20260927 00:43:18.751946 21603 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:18.752063 21613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:18.752345 21613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc72401573b4ed09d731867d5cedfc6, State: Running, Role: LEADER
I20260927 00:43:18.752656 21613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } }
I20260927 00:43:18.753157 21603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:18.754396 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc72401573b4ed09d731867d5cedfc6. Latest consensus state: current_term: 1 leader_uuid: "8dc72401573b4ed09d731867d5cedfc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } } }
I20260927 00:43:18.754376 21614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc72401573b4ed09d731867d5cedfc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc72401573b4ed09d731867d5cedfc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44383 } } }
I20260927 00:43:18.754531 21614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:18.754531 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:18.754997 21623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:18.758378 21623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:18.764631 21623 catalog_manager.cc:1347] Generated new cluster ID: c5f5d24b379e43fc9b9e52997f887dcf
I20260927 00:43:18.764710 21623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:18.785492 21623 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:18.785852 21623 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:18.793815 21623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc72401573b4ed09d731867d5cedfc6: Generated new TSK 0
I20260927 00:43:18.794739 21623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:18.809265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21535
2026-09-27T00:43:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:18 dist-test-slave-xltq krb5kdc[21634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:18 dist-test-slave-xltq krb5kdc[21634](info): set up 2 sockets
Sep 27 00:43:18 dist-test-slave-xltq krb5kdc[21634](info): commencing operation
krb5kdc: starting...
W20260927 00:43:19.768318 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:19 dist-test-slave-xltq krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:19.894903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:20.007337 21647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:20.007640 21647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:20.007684 21647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:20.011970 21647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:20.012056 21647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:20.012075 21647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:20.012091 21647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:20.012108 21647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:20.017771 21647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:20.018898 21647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:20.019914 21647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:20.028404 21652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:20.028410 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:20.028404 21653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:20.028697 21647 server_base.cc:1034] running on GCE node
I20260927 00:43:20.029412 21647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:20.030773 21647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:20.031965 21647 hybrid_clock.cc:648] HybridClock initialized: now 1790469800031944 us; error 36 us; skew 500 ppm
Sep 27 00:43:20 dist-test-slave-xltq krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:20.035058 21647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:20.036022 21647 webserver.cc:460] Webserver started at http://127.0.0.1:36235/ using document root <none> and password file <none>
I20260927 00:43:20.036612 21647 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:20.036679 21647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:20.036885 21647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:20.039006 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "3065f7887ff947439cc96f3059793304"
format_stamp: "Formatted at 2026-09-27 00:43:20 on dist-test-slave-xltq"
server_key: "332091ebc81ddfdd68f860ffbf447ba5"
server_key_iv: "7c48019fc054595c48260db938b551b6"
server_key_version: "encryptionkey@0"
I20260927 00:43:20.039597 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "3065f7887ff947439cc96f3059793304"
format_stamp: "Formatted at 2026-09-27 00:43:20 on dist-test-slave-xltq"
server_key: "332091ebc81ddfdd68f860ffbf447ba5"
server_key_iv: "7c48019fc054595c48260db938b551b6"
server_key_version: "encryptionkey@0"
I20260927 00:43:20.043502 21647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:20.046105 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:20.047473 21647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:20.047602 21647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "3065f7887ff947439cc96f3059793304"
format_stamp: "Formatted at 2026-09-27 00:43:20 on dist-test-slave-xltq"
server_key: "332091ebc81ddfdd68f860ffbf447ba5"
server_key_iv: "7c48019fc054595c48260db938b551b6"
server_key_version: "encryptionkey@0"
I20260927 00:43:20.047714 21647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:20.058355 21647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:20.061803 21647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:20.062000 21647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:20.069010 21647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35347
I20260927 00:43:20.069031 21714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35347 every 8 connection(s)
I20260927 00:43:20.069952 21647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:20.071183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21647
I20260927 00:43:20.071308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:20.071583 392 external_mini_cluster.cc:1428] Setting key 190abbc1e237f5f742d24ad5956e518f
I20260927 00:43:20.073078 21715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:20.078742 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304: Bootstrap starting.
Sep 27 00:43:20 dist-test-slave-xltq krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:43:20.081291 21715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:20.082101 21715 log.cc:824] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:20.084461 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304: No bootstrap required, opened a new log
I20260927 00:43:20.085785 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:20.072939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:58633 (local address 127.0.0.1:35347)
0927 00:43:20.073386 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:20.073423 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:20.073460 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:20.074096 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:20.074113 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:20.074465 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:20.074767 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:20.075583 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:20.076624 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:20.077450 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:20.077908 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:20.078049 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:20.080802 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:20.080807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:20.080821 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:20.080850 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:20.082623 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:20.083336 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:20.083341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:20.083345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:20.083465 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:20.083787 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:20.083789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:20.083790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:20.084280 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:20.084440 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:20.084760 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:43:20.084987 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260927 00:43:20.088630 21715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } }
I20260927 00:43:20.088848 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:20.088919 21715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:20.088973 21715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065f7887ff947439cc96f3059793304, State: Initialized, Role: FOLLOWER
I20260927 00:43:20.089442 21715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } }
I20260927 00:43:20.089596 21715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:20.089665 21715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:20.089764 21715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:20.091007 21715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } }
I20260927 00:43:20.091388 21715 leader_election.cc:302] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3065f7887ff947439cc96f3059793304; no voters:
I20260927 00:43:20.091739 21715 leader_election.cc:288] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:20.091815 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:20.092044 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [term 1 LEADER]: Becoming Leader. State: Replica: 3065f7887ff947439cc96f3059793304, State: Running, Role: LEADER
I20260927 00:43:20.092384 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } }
I20260927 00:43:20.093206 21715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:20.094116 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3065f7887ff947439cc96f3059793304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } } }
I20260927 00:43:20.094278 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:20.094786 21729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:20.095091 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3065f7887ff947439cc96f3059793304. Latest consensus state: current_term: 1 leader_uuid: "3065f7887ff947439cc96f3059793304" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f7887ff947439cc96f3059793304" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35347 } } }
I20260927 00:43:20.095216 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:20.098446 21729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:20.101481 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:20.091052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:37243 (local address 127.0.0.1:35347)
0927 00:43:20.091177 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:20.091182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:20.091474 (+ 292us) server_negotiation.cc:407] Connection header received
0927 00:43:20.091665 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:20.091668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:20.091725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:20.091820 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:20.092744 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:20.093383 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:20.094591 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:20.094935 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:20.095041 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:20.097059 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:20.097063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:20.097066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:20.097097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:20.098819 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:20.099728 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:20.099732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:20.099734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:20.099838 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:20.100371 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:20.100375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:20.100377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:20.100620 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:20.100785 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:20.101203 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 00:43:20.101306 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260927 00:43:20.105120 21729 catalog_manager.cc:1347] Generated new cluster ID: eb86b0154833401e903ef59800088d53
I20260927 00:43:20.105199 21729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:20.122879 21729 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:20.124006 21729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:20.130831 21729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3065f7887ff947439cc96f3059793304: Generated new TSK 0
I20260927 00:43:20.131752 21729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:20.137084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21647
W20260927 00:43:20.141649 21632 connection.cc:535] client connection to 127.0.0.1:35347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35347): Connection reset by peer (error 104)
2026-09-27T00:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:20 dist-test-slave-xltq krb5kdc[21746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:20 dist-test-slave-xltq krb5kdc[21746](info): set up 2 sockets
Sep 27 00:43:20 dist-test-slave-xltq krb5kdc[21746](info): commencing operation
krb5kdc: starting...
W20260927 00:43:21.092120 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:21 dist-test-slave-xltq krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:21.215425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:21.327652 21759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:21.327939 21759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:21.327980 21759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:21.332183 21759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:21.332279 21759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:21.332299 21759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:21.332315 21759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:21.332329 21759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:21.337944 21759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:21.339084 21759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:21.340121 21759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:21.349078 21765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:21.349077 21764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:21.349328 21759 server_base.cc:1034] running on GCE node
W20260927 00:43:21.349105 21767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:21.350121 21759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:21.351323 21759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:21.352494 21759 hybrid_clock.cc:648] HybridClock initialized: now 1790469801352480 us; error 31 us; skew 500 ppm
Sep 27 00:43:21 dist-test-slave-xltq krb5kdc[21746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:21.355327 21759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:21.356362 21759 webserver.cc:460] Webserver started at http://127.0.0.1:35137/ using document root <none> and password file <none>
I20260927 00:43:21.356933 21759 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:21.356999 21759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:21.357173 21759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:21.359369 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "5bd6caeccde04b4186ecb067c09f147a"
format_stamp: "Formatted at 2026-09-27 00:43:21 on dist-test-slave-xltq"
server_key: "c9695db26d7d9b2f4f9040fef712e578"
server_key_iv: "26f9b7f8636d6ea1afd8f8306a4a02d5"
server_key_version: "encryptionkey@0"
I20260927 00:43:21.359961 21759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bd6caeccde04b4186ecb067c09f147a"
format_stamp: "Formatted at 2026-09-27 00:43:21 on dist-test-slave-xltq"
server_key: "c9695db26d7d9b2f4f9040fef712e578"
server_key_iv: "26f9b7f8636d6ea1afd8f8306a4a02d5"
server_key_version: "encryptionkey@0"
I20260927 00:43:21.364022 21759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 00:43:21.366694 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:21.368070 21759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:21.368187 21759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "5bd6caeccde04b4186ecb067c09f147a"
format_stamp: "Formatted at 2026-09-27 00:43:21 on dist-test-slave-xltq"
server_key: "c9695db26d7d9b2f4f9040fef712e578"
server_key_iv: "26f9b7f8636d6ea1afd8f8306a4a02d5"
server_key_version: "encryptionkey@0"
I20260927 00:43:21.368336 21759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:21.389943 21759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:21.393213 21759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:21.393392 21759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:21.400069 21759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36145
I20260927 00:43:21.400110 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36145 every 8 connection(s)
I20260927 00:43:21.401077 21759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:21.402189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21759
I20260927 00:43:21.402334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:21.402638 392 external_mini_cluster.cc:1428] Setting key e34377984757b10565ba6ad4dd38cf52
I20260927 00:43:21.404548 21827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:21.411216 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: Bootstrap starting.
I20260927 00:43:21.411953 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:21.404041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36073 (local address 127.0.0.1:36145)
0927 00:43:21.404468 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:21.404484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:21.404522 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:43:21.405208 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:21.405226 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:21.405596 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:21.405904 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:21.406236 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:21.407249 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:21.408529 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:21.409040 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:21.409170 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:21.410087 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:21.410092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:21.410106 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:21.410516 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:21.410575 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:21.410599 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:21.410701 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:21.410903 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 00:43:21.411048 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20260927 00:43:21.413532 21827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:21.414320 21827 log.cc:824] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:21.414841 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:21.416688 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: No bootstrap required, opened a new log
I20260927 00:43:21.420387 21827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260927 00:43:21.420702 21827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:21.420766 21827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd6caeccde04b4186ecb067c09f147a, State: Initialized, Role: FOLLOWER
I20260927 00:43:21.421190 21827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260927 00:43:21.421322 21827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:21.421363 21827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:21.421455 21827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:21.422386 21827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260927 00:43:21.422714 21827 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd6caeccde04b4186ecb067c09f147a; no voters:
I20260927 00:43:21.423038 21827 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:21.423128 21837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:21.423632 21837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [term 1 LEADER]: Becoming Leader. State: Replica: 5bd6caeccde04b4186ecb067c09f147a, State: Running, Role: LEADER
I20260927 00:43:21.423960 21837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } }
I20260927 00:43:21.424338 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:21.417047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:47561 (local address 127.0.0.1:36145)
0927 00:43:21.417167 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:21.417171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:21.420533 (+ 3362us) server_negotiation.cc:407] Connection header received
0927 00:43:21.420675 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:21.420678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:21.420740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:21.420843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:21.421145 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:21.421735 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:21.422584 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:21.422798 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:21.422888 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:21.423795 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:21.423798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:21.423801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:21.423815 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:21.423842 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:21.423855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:21.423913 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:21.424136 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 00:43:21.424179 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:43:21.424456 21827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:21.425905 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd6caeccde04b4186ecb067c09f147a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } } }
I20260927 00:43:21.426024 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:21.427030 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd6caeccde04b4186ecb067c09f147a. Latest consensus state: current_term: 1 leader_uuid: "5bd6caeccde04b4186ecb067c09f147a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd6caeccde04b4186ecb067c09f147a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36145 } } }
I20260927 00:43:21.427130 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a [sys.catalog]: This master's current role is: LEADER
W20260927 00:43:21.429605 21852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:21.429687 21852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:21.429819 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:21.430743 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:21.437054 21848 catalog_manager.cc:1347] Generated new cluster ID: 84cc183e8ef54f8ba30414946d58e572
I20260927 00:43:21.437129 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:21.452287 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:21.453279 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:21.459453 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd6caeccde04b4186ecb067c09f147a: Generated new TSK 0
I20260927 00:43:21.460177 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:21.472620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21759
W20260927 00:43:21.479018 21744 connection.cc:535] client connection to 127.0.0.1:36145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36145): Connection reset by peer (error 104)
2026-09-27T00:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:21 dist-test-slave-xltq krb5kdc[21858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:21 dist-test-slave-xltq krb5kdc[21858](info): set up 2 sockets
Sep 27 00:43:21 dist-test-slave-xltq krb5kdc[21858](info): commencing operation
krb5kdc: starting...
W20260927 00:43:22.440280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:22 dist-test-slave-xltq krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:43:22.567174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:22.681596 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:22.681918 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:22.681962 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:22.686545 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:22.686657 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:22.686682 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:22.686699 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:22.686745 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:22.693297 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:22.694540 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:22.695616 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:22.704823 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:22.704888 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:22.704823 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:22.705121 21871 server_base.cc:1034] running on GCE node
I20260927 00:43:22.706030 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:22.707286 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:22.708460 21871 hybrid_clock.cc:648] HybridClock initialized: now 1790469802708433 us; error 50 us; skew 500 ppm
Sep 27 00:43:22 dist-test-slave-xltq krb5kdc[21858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:22.715174 21871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:43:22.716482 21871 webserver.cc:460] Webserver started at http://127.0.0.1:44489/ using document root <none> and password file <none>
I20260927 00:43:22.717231 21871 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:22.717396 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:22.717658 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:22.719955 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "af3d5d979ea548d9877ce5ef3aac11c9"
format_stamp: "Formatted at 2026-09-27 00:43:22 on dist-test-slave-xltq"
server_key: "18674f78180932494735e6153d8bec0e"
server_key_iv: "a36aaa1e67854378bcdff7c99909dc2d"
server_key_version: "encryptionkey@0"
I20260927 00:43:22.720672 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "af3d5d979ea548d9877ce5ef3aac11c9"
format_stamp: "Formatted at 2026-09-27 00:43:22 on dist-test-slave-xltq"
server_key: "18674f78180932494735e6153d8bec0e"
server_key_iv: "a36aaa1e67854378bcdff7c99909dc2d"
server_key_version: "encryptionkey@0"
I20260927 00:43:22.725320 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 00:43:22.728081 21886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:22.729542 21871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:22.729696 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "af3d5d979ea548d9877ce5ef3aac11c9"
format_stamp: "Formatted at 2026-09-27 00:43:22 on dist-test-slave-xltq"
server_key: "18674f78180932494735e6153d8bec0e"
server_key_iv: "a36aaa1e67854378bcdff7c99909dc2d"
server_key_version: "encryptionkey@0"
I20260927 00:43:22.729826 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:22.735183 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:22.738936 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:22.739149 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:22.746924 21871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36663
I20260927 00:43:22.746939 21938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36663 every 8 connection(s)
I20260927 00:43:22.747900 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:22.750913 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:22.754025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21871
I20260927 00:43:22.754177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:22.754482 392 external_mini_cluster.cc:1428] Setting key 324d6552322318636d1fcc3f17a1c624
I20260927 00:43:22.759003 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9: Bootstrap starting.
I20260927 00:43:22.760847 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:22.756341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:41865 (local address 127.0.0.1:36663)
0927 00:43:22.756883 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:22.756899 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:22.756939 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:43:22.757783 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:22.757806 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:22.758134 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:22.758506 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:22.759564 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:22.759570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:22.759587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:22.759609 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:22.759675 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:22.759697 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:22.759829 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:22.759901 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:43:22.760065 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":171,"threads_started":1}
I20260927 00:43:22.761722 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:22.762439 21939 log.cc:824] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:22.763898 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:22.764601 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9: No bootstrap required, opened a new log
I20260927 00:43:22.767838 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:22.766110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49473 (local address 127.0.0.1:36663)
0927 00:43:22.766262 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:22.766267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:22.766446 (+ 179us) server_negotiation.cc:407] Connection header received
0927 00:43:22.766580 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:22.766583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:22.766620 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:22.766680 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:22.767458 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:22.767461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:22.767464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:22.767476 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:22.767504 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:22.767515 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:22.767592 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:22.767659 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:43:22.767704 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:43:22.768700 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } }
I20260927 00:43:22.769014 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:22.769070 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3d5d979ea548d9877ce5ef3aac11c9, State: Initialized, Role: FOLLOWER
I20260927 00:43:22.769462 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } }
I20260927 00:43:22.769665 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:22.769855 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:22.769984 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:22.771135 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } }
I20260927 00:43:22.771469 21939 leader_election.cc:302] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3d5d979ea548d9877ce5ef3aac11c9; no voters:
I20260927 00:43:22.771813 21939 leader_election.cc:288] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:22.771919 21949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:22.772130 21949 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [term 1 LEADER]: Becoming Leader. State: Replica: af3d5d979ea548d9877ce5ef3aac11c9, State: Running, Role: LEADER
I20260927 00:43:22.772493 21949 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } }
I20260927 00:43:22.773221 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:22.774514 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3d5d979ea548d9877ce5ef3aac11c9. Latest consensus state: current_term: 1 leader_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } } }
I20260927 00:43:22.774561 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3d5d979ea548d9877ce5ef3aac11c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36663 } } }
I20260927 00:43:22.774650 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:22.774678 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:22.775110 21959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:22.778692 21959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:22.785485 21959 catalog_manager.cc:1347] Generated new cluster ID: f4042dbabf5641a98cf9498952b28795
I20260927 00:43:22.785593 21959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:22.794945 21959 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:22.795363 21959 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:22.809988 21959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3d5d979ea548d9877ce5ef3aac11c9: Generated new TSK 0
I20260927 00:43:22.810811 21959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:22.818511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21871
2026-09-27T00:43:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:22 dist-test-slave-xltq krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:22 dist-test-slave-xltq krb5kdc[21970](info): set up 2 sockets
Sep 27 00:43:22 dist-test-slave-xltq krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260927 00:43:23.792304 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:23 dist-test-slave-xltq krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:23.915588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:24.028676 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:24.028980 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:24.029023 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:24.033239 21983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:24.033313 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:24.033334 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:24.033351 21983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:24.033365 21983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:24.038921 21983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:24.040005 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:24.041028 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:43:24.050109 21983 server_base.cc:1034] running on GCE node
W20260927 00:43:24.049926 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:24.049950 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:24.050274 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:24.050947 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:24.052161 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:24.053350 21983 hybrid_clock.cc:648] HybridClock initialized: now 1790469804053328 us; error 40 us; skew 500 ppm
Sep 27 00:43:24 dist-test-slave-xltq krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:24.056900 21983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:24.058248 21983 webserver.cc:460] Webserver started at http://127.0.98.62:33059/ using document root <none> and password file <none>
I20260927 00:43:24.059203 21983 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:24.059288 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:24.059516 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:24.062816 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "d193edeb9ef7486484336bb19c2d8297"
format_stamp: "Formatted at 2026-09-27 00:43:24 on dist-test-slave-xltq"
server_key: "f456bda0b9d00713b67537c0af7a48eb"
server_key_iv: "0b0e5fcdcf7e95d525ac038aa230f1f5"
server_key_version: "encryptionkey@0"
I20260927 00:43:24.063825 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "d193edeb9ef7486484336bb19c2d8297"
format_stamp: "Formatted at 2026-09-27 00:43:24 on dist-test-slave-xltq"
server_key: "f456bda0b9d00713b67537c0af7a48eb"
server_key_iv: "0b0e5fcdcf7e95d525ac038aa230f1f5"
server_key_version: "encryptionkey@0"
I20260927 00:43:24.070704 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 00:43:24.074795 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:24.076653 21983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:43:24.076894 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "d193edeb9ef7486484336bb19c2d8297"
format_stamp: "Formatted at 2026-09-27 00:43:24 on dist-test-slave-xltq"
server_key: "f456bda0b9d00713b67537c0af7a48eb"
server_key_iv: "0b0e5fcdcf7e95d525ac038aa230f1f5"
server_key_version: "encryptionkey@0"
I20260927 00:43:24.077087 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:24.122915 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:24.126266 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:24.126436 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:24.133529 21983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36791
I20260927 00:43:24.133584 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36791 every 8 connection(s)
I20260927 00:43:24.134543 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:24.137537 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:24.143020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 21983
I20260927 00:43:24.143138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:24.143240 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297: Bootstrap starting.
I20260927 00:43:24.143378 392 external_mini_cluster.cc:1428] Setting key de7c978a93fa2d399c5f1dea855062c1
I20260927 00:43:24.145826 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:24.146556 22051 log.cc:824] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:24.148607 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297: No bootstrap required, opened a new log
Sep 27 00:43:24 dist-test-slave-xltq krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:24.153167 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } }
I20260927 00:43:24.153472 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:24.153541 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d193edeb9ef7486484336bb19c2d8297, State: Initialized, Role: FOLLOWER
I20260927 00:43:24.154036 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } }
I20260927 00:43:24.154206 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:24.154275 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:24.154383 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:24.155506 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } }
I20260927 00:43:24.155906 22051 leader_election.cc:302] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d193edeb9ef7486484336bb19c2d8297; no voters:
I20260927 00:43:24.156297 22051 leader_election.cc:288] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:24.156392 22056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:24.156618 22056 raft_consensus.cc:686] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [term 1 LEADER]: Becoming Leader. State: Replica: d193edeb9ef7486484336bb19c2d8297, State: Running, Role: LEADER
I20260927 00:43:24.156927 22056 consensus_queue.cc:237] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } }
I20260927 00:43:24.157709 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:24.158633 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d193edeb9ef7486484336bb19c2d8297. Latest consensus state: current_term: 1 leader_uuid: "d193edeb9ef7486484336bb19c2d8297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } } }
I20260927 00:43:24.158768 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:24.159111 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d193edeb9ef7486484336bb19c2d8297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d193edeb9ef7486484336bb19c2d8297" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36791 } } }
I20260927 00:43:24.159183 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:24.159209 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:24.160156 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:24.144739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59441 (local address 127.0.98.62:36791)
0927 00:43:24.145159 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:24.145174 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:24.145211 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:24.145874 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:24.145891 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:24.146272 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:24.146547 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:24.147390 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:24.148514 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:24.149674 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:24.150179 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:24.150303 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:24.153397 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:24.153403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:24.153424 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:24.153463 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:24.155656 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:24.156438 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:24.156444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:24.156450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:24.156644 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:24.157198 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:24.157202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:24.157203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:24.157712 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:24.158010 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:24.158524 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 00:43:24.158891 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260927 00:43:24.162578 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:24.165021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:24.169507 22065 catalog_manager.cc:1347] Generated new cluster ID: 9d1fad8c08a6400a8acd2da8234677fd
I20260927 00:43:24.169596 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:24.183902 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:24.167261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57689 (local address 127.0.98.62:36791)
0927 00:43:24.167367 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:24.167371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:24.173050 (+ 5679us) server_negotiation.cc:407] Connection header received
0927 00:43:24.173127 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:24.173131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:24.173188 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:24.173270 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:24.174387 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:24.175252 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:24.176494 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:24.176868 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:24.176949 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:24.179960 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:24.179965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:24.179968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:24.180003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:24.181696 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:24.182434 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:24.182439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:24.182441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:24.182525 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:24.182880 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:24.182883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:24.182885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:24.183113 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:24.183229 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:24.183625 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 00:43:24.183730 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:43:24.205425 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:24.206549 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:24.213727 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d193edeb9ef7486484336bb19c2d8297: Generated new TSK 0
I20260927 00:43:24.214495 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:24.221055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 21983
W20260927 00:43:24.226065 21968 connection.cc:535] client connection to 127.0.98.62:36791 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36791): Connection reset by peer (error 104)
2026-09-27T00:43:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:24 dist-test-slave-xltq krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:24 dist-test-slave-xltq krb5kdc[22082](info): set up 2 sockets
Sep 27 00:43:24 dist-test-slave-xltq krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260927 00:43:25.188256 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:25 dist-test-slave-xltq krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:25.312047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:25.422646 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:25.422928 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:25.422984 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:25.427104 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:25.427181 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:25.427209 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:25.427230 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:25.427258 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:25.432847 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:25.433945 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:25.434947 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:25.443436 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:25.443436 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:25.443598 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:25.443830 22095 server_base.cc:1034] running on GCE node
I20260927 00:43:25.444423 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:25.445510 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:25.446676 22095 hybrid_clock.cc:648] HybridClock initialized: now 1790469805446642 us; error 43 us; skew 500 ppm
Sep 27 00:43:25 dist-test-slave-xltq krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:25.449563 22095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:25.450489 22095 webserver.cc:460] Webserver started at http://127.0.98.62:35175/ using document root <none> and password file <none>
I20260927 00:43:25.451030 22095 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:25.451088 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:25.451278 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:25.453410 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "70eff0cf4998403291ccec10b9b853f0"
format_stamp: "Formatted at 2026-09-27 00:43:25 on dist-test-slave-xltq"
server_key: "b383fd4a92addb00f344bc84af6fc6e5"
server_key_iv: "06bbaad8a60295e4ece27811c62a84f6"
server_key_version: "encryptionkey@0"
I20260927 00:43:25.453991 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "70eff0cf4998403291ccec10b9b853f0"
format_stamp: "Formatted at 2026-09-27 00:43:25 on dist-test-slave-xltq"
server_key: "b383fd4a92addb00f344bc84af6fc6e5"
server_key_iv: "06bbaad8a60295e4ece27811c62a84f6"
server_key_version: "encryptionkey@0"
I20260927 00:43:25.457950 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:25.460506 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:25.461786 22095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:43:25.461907 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "70eff0cf4998403291ccec10b9b853f0"
format_stamp: "Formatted at 2026-09-27 00:43:25 on dist-test-slave-xltq"
server_key: "b383fd4a92addb00f344bc84af6fc6e5"
server_key_iv: "06bbaad8a60295e4ece27811c62a84f6"
server_key_version: "encryptionkey@0"
I20260927 00:43:25.462023 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:25.512318 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:25.515666 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:25.515861 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:25.522825 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44265
I20260927 00:43:25.523140 22162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44265 every 8 connection(s)
I20260927 00:43:25.523944 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:25.527014 22163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:25.529064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22095
I20260927 00:43:25.529196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:25.529449 392 external_mini_cluster.cc:1428] Setting key 99a9d760b887f12ad96e96ae8545eccf
I20260927 00:43:25.534328 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0: Bootstrap starting.
I20260927 00:43:25.537384 22163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:25.538296 22163 log.cc:824] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:25.540395 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:25.530917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37319 (local address 127.0.98.62:44265)
0927 00:43:25.531392 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:25.531413 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:25.531461 (+ 48us) server_negotiation.cc:407] Connection header received
0927 00:43:25.532260 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:25.532285 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:25.532768 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:25.533171 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:25.533604 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:25.535222 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:25.536361 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:25.537102 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:25.537264 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:25.538160 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:25.538166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:25.538185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:25.538704 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:25.538776 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:25.538806 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:25.538920 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:25.539224 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 00:43:25.539368 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260927 00:43:25.540874 22163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0: No bootstrap required, opened a new log
I20260927 00:43:25.543586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:25.544948 22163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } }
I20260927 00:43:25.545193 22163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:25.545229 22163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70eff0cf4998403291ccec10b9b853f0, State: Initialized, Role: FOLLOWER
I20260927 00:43:25.545677 22163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } }
I20260927 00:43:25.545836 22163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:25.545912 22163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:25.546015 22163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:25.547209 22163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } }
I20260927 00:43:25.547577 22163 leader_election.cc:302] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70eff0cf4998403291ccec10b9b853f0; no voters:
I20260927 00:43:25.547897 22163 leader_election.cc:288] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:25.548003 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:25.548247 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [term 1 LEADER]: Becoming Leader. State: Replica: 70eff0cf4998403291ccec10b9b853f0, State: Running, Role: LEADER
I20260927 00:43:25.548578 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } }
I20260927 00:43:25.549312 22163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:25.549662 22173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70eff0cf4998403291ccec10b9b853f0. Latest consensus state: current_term: 1 leader_uuid: "70eff0cf4998403291ccec10b9b853f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } } }
I20260927 00:43:25.549774 22173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:25.550088 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70eff0cf4998403291ccec10b9b853f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70eff0cf4998403291ccec10b9b853f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44265 } } }
I20260927 00:43:25.550195 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:25.550202 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:25.545560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55369 (local address 127.0.98.62:44265)
0927 00:43:25.545682 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:25.545686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:25.545856 (+ 170us) server_negotiation.cc:407] Connection header received
0927 00:43:25.545975 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:25.545978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:25.546030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:25.546115 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:25.546355 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:25.547102 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:25.548046 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:25.548387 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:25.548482 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:25.549528 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:25.549531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:25.549534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:25.549547 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:25.549576 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:25.549589 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:25.549663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:25.549988 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 00:43:25.550024 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:43:25.550375 22178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:25.553516 22178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:25.559321 22178 catalog_manager.cc:1347] Generated new cluster ID: 4eab4dbbb89d41dba3cf8524e583ba92
I20260927 00:43:25.559412 22178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:25.591610 22178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:25.592708 22178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:25.599496 22178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70eff0cf4998403291ccec10b9b853f0: Generated new TSK 0
I20260927 00:43:25.600262 22178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:25.605264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22095
W20260927 00:43:25.610170 22080 connection.cc:535] client connection to 127.0.98.62:44265 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44265): Connection reset by peer (error 104)
2026-09-27T00:43:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:25 dist-test-slave-xltq krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:25 dist-test-slave-xltq krb5kdc[22194](info): set up 2 sockets
Sep 27 00:43:25 dist-test-slave-xltq krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260927 00:43:26.564436 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:26 dist-test-slave-xltq krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:26.688755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:26.802729 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:26.803049 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:26.803094 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:26.807331 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:26.807416 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:26.807435 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:26.807451 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:26.807466 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:26.813354 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:26.814566 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:26.815639 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:26.824293 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:26.824394 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:26.824365 22207 server_base.cc:1034] running on GCE node
W20260927 00:43:26.824293 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:26.825277 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:26.826506 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:26.827762 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790469806827739 us; error 44 us; skew 500 ppm
Sep 27 00:43:26 dist-test-slave-xltq krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:26.830726 22207 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:26.831779 22207 webserver.cc:460] Webserver started at http://127.0.98.62:43375/ using document root <none> and password file <none>
I20260927 00:43:26.832373 22207 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:26.832444 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:26.832686 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:26.834971 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "eacadeb864dd4b368c6c939394453fcd"
format_stamp: "Formatted at 2026-09-27 00:43:26 on dist-test-slave-xltq"
server_key: "0189752552ba665e84c74e0ffe001b67"
server_key_iv: "88f33f05023982b7b3453d4d1871efee"
server_key_version: "encryptionkey@0"
I20260927 00:43:26.835537 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "eacadeb864dd4b368c6c939394453fcd"
format_stamp: "Formatted at 2026-09-27 00:43:26 on dist-test-slave-xltq"
server_key: "0189752552ba665e84c74e0ffe001b67"
server_key_iv: "88f33f05023982b7b3453d4d1871efee"
server_key_version: "encryptionkey@0"
I20260927 00:43:26.839439 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:43:26.842090 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:26.843502 22207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:26.843637 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "eacadeb864dd4b368c6c939394453fcd"
format_stamp: "Formatted at 2026-09-27 00:43:26 on dist-test-slave-xltq"
server_key: "0189752552ba665e84c74e0ffe001b67"
server_key_iv: "88f33f05023982b7b3453d4d1871efee"
server_key_version: "encryptionkey@0"
I20260927 00:43:26.843761 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:26.848565 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:26.851794 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:26.851985 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:26.859203 22207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45263
I20260927 00:43:26.859218 22274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45263 every 8 connection(s)
I20260927 00:43:26.860138 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:26.863001 22275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:26.865557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22207
I20260927 00:43:26.865676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:26.865944 392 external_mini_cluster.cc:1428] Setting key 2ba35f0f78904c74aeed6425d42a314d
I20260927 00:43:26.868841 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd: Bootstrap starting.
I20260927 00:43:26.870891 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:26.867294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34081 (local address 127.0.98.62:45263)
0927 00:43:26.867690 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:26.867705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:26.867747 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:43:26.868417 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:26.868435 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:26.868713 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:26.869038 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:26.869766 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:26.869771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:26.869782 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:26.869792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:26.869844 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:26.869863 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:26.869982 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:26.870049 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:43:26.870202 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260927 00:43:26.871202 22275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:26.871951 22275 log.cc:824] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:26.873895 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:26.874137 22275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd: No bootstrap required, opened a new log
I20260927 00:43:26.877468 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:26.875893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42327 (local address 127.0.98.62:45263)
0927 00:43:26.876005 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:26.876009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:26.876146 (+ 137us) server_negotiation.cc:407] Connection header received
0927 00:43:26.876299 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:26.876302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:26.876338 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:26.876412 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:26.877105 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:26.877108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:26.877111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:26.877123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:26.877151 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:26.877162 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:26.877231 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:26.877301 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 00:43:26.877343 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:43:26.878499 22275 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } }
I20260927 00:43:26.878794 22275 raft_consensus.cc:374] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:26.878854 22275 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacadeb864dd4b368c6c939394453fcd, State: Initialized, Role: FOLLOWER
I20260927 00:43:26.879287 22275 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } }
I20260927 00:43:26.879431 22275 raft_consensus.cc:388] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:26.879470 22275 raft_consensus.cc:482] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:26.879567 22275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:26.880635 22275 raft_consensus.cc:504] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } }
I20260927 00:43:26.880988 22275 leader_election.cc:302] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eacadeb864dd4b368c6c939394453fcd; no voters:
I20260927 00:43:26.881331 22275 leader_election.cc:288] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:26.881419 22285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:26.881635 22285 raft_consensus.cc:686] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [term 1 LEADER]: Becoming Leader. State: Replica: eacadeb864dd4b368c6c939394453fcd, State: Running, Role: LEADER
I20260927 00:43:26.882035 22285 consensus_queue.cc:237] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } }
I20260927 00:43:26.882608 22275 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:26.883617 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eacadeb864dd4b368c6c939394453fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } } }
I20260927 00:43:26.883744 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:26.884020 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader eacadeb864dd4b368c6c939394453fcd. Latest consensus state: current_term: 1 leader_uuid: "eacadeb864dd4b368c6c939394453fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacadeb864dd4b368c6c939394453fcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45263 } } }
I20260927 00:43:26.884116 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:26.884100 22294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:26.887950 22294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:26.894183 22294 catalog_manager.cc:1347] Generated new cluster ID: 29df9f033a0544f1a948e60b251bdccf
I20260927 00:43:26.894289 22294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:26.929641 22294 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:26.929987 22294 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:26.939388 22294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eacadeb864dd4b368c6c939394453fcd: Generated new TSK 0
I20260927 00:43:26.940155 22294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:26.951344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22207
2026-09-27T00:43:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:27 dist-test-slave-xltq krb5kdc[22306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:27 dist-test-slave-xltq krb5kdc[22306](info): set up 2 sockets
Sep 27 00:43:27 dist-test-slave-xltq krb5kdc[22306](info): commencing operation
krb5kdc: starting...
W20260927 00:43:27.904424 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:27 dist-test-slave-xltq krb5kdc[22306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:28.030751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:28.143811 22319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:28.144129 22319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:28.144191 22319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:28.148350 22319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:28.148423 22319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:28.148442 22319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:28.148458 22319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:28.148471 22319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:28.153919 22319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:28.155047 22319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:28.156018 22319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:28.164623 22324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:28.164645 22325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:28.164615 22327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:28.164685 22319 server_base.cc:1034] running on GCE node
I20260927 00:43:28.165472 22319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:28.166697 22319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:28.167902 22319 hybrid_clock.cc:648] HybridClock initialized: now 1790469808167827 us; error 88 us; skew 500 ppm
Sep 27 00:43:28 dist-test-slave-xltq krb5kdc[22306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:28.176506 22319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:28.177597 22319 webserver.cc:460] Webserver started at http://127.0.98.62:36115/ using document root <none> and password file <none>
I20260927 00:43:28.178190 22319 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:28.178265 22319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:28.178474 22319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:28.180701 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "df304638e1be4a2ca9c84aa6592e8e8f"
format_stamp: "Formatted at 2026-09-27 00:43:28 on dist-test-slave-xltq"
server_key: "c18f3a54263b7412a011205513cf74d1"
server_key_iv: "7fadd1ea862c3c62015d39a4a8a9109a"
server_key_version: "encryptionkey@0"
I20260927 00:43:28.181265 22319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "df304638e1be4a2ca9c84aa6592e8e8f"
format_stamp: "Formatted at 2026-09-27 00:43:28 on dist-test-slave-xltq"
server_key: "c18f3a54263b7412a011205513cf74d1"
server_key_iv: "7fadd1ea862c3c62015d39a4a8a9109a"
server_key_version: "encryptionkey@0"
I20260927 00:43:28.185117 22319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:28.187847 22334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:28.189419 22319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:28.189570 22319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "df304638e1be4a2ca9c84aa6592e8e8f"
format_stamp: "Formatted at 2026-09-27 00:43:28 on dist-test-slave-xltq"
server_key: "c18f3a54263b7412a011205513cf74d1"
server_key_iv: "7fadd1ea862c3c62015d39a4a8a9109a"
server_key_version: "encryptionkey@0"
I20260927 00:43:28.189706 22319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:28.209684 22319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:28.213353 22319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:28.213568 22319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:28.220902 22319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38585
I20260927 00:43:28.220979 22386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38585 every 8 connection(s)
I20260927 00:43:28.221940 22319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:28.224851 22387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:28.227707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22319
I20260927 00:43:28.227833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:28.228088 392 external_mini_cluster.cc:1428] Setting key eba5107e0c115e388a3b0a7f39e55efb
I20260927 00:43:28.230906 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f: Bootstrap starting.
I20260927 00:43:28.233160 22387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:28.233906 22387 log.cc:824] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:28.235999 22387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f: No bootstrap required, opened a new log
Sep 27 00:43:28 dist-test-slave-xltq krb5kdc[22306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:28.239653 22387 raft_consensus.cc:348] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260927 00:43:28.239960 22387 raft_consensus.cc:374] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:28.240001 22387 raft_consensus.cc:729] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df304638e1be4a2ca9c84aa6592e8e8f, State: Initialized, Role: FOLLOWER
I20260927 00:43:28.240505 22387 consensus_queue.cc:260] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260927 00:43:28.240639 22387 raft_consensus.cc:388] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:28.240703 22387 raft_consensus.cc:482] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:28.240803 22387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:28.241765 22387 raft_consensus.cc:504] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260927 00:43:28.242179 22387 leader_election.cc:302] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df304638e1be4a2ca9c84aa6592e8e8f; no voters:
I20260927 00:43:28.242600 22387 leader_election.cc:288] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:28.242722 22392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:28.243170 22392 raft_consensus.cc:686] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [term 1 LEADER]: Becoming Leader. State: Replica: df304638e1be4a2ca9c84aa6592e8e8f, State: Running, Role: LEADER
I20260927 00:43:28.243623 22392 consensus_queue.cc:237] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } }
I20260927 00:43:28.244251 22387 sys_catalog.cc:564] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:28.244376 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:28.229542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45083 (local address 127.0.98.62:38585)
0927 00:43:28.230107 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:28.230123 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:28.230160 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:28.230844 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:28.230862 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:28.231347 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:28.231701 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:28.232522 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:28.233631 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:28.234509 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:28.235036 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:28.235204 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:28.238144 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:28.238150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:28.238179 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:28.238215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:28.240836 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:28.241336 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:28.241343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:28.241350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:28.241483 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:28.241768 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:28.241772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:28.241774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:28.242282 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:28.242478 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:28.242948 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 00:43:28.243288 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":168,"threads_started":1}
I20260927 00:43:28.244870 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df304638e1be4a2ca9c84aa6592e8e8f. Latest consensus state: current_term: 1 leader_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260927 00:43:28.245575 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:28.245462 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df304638e1be4a2ca9c84aa6592e8e8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38585 } } }
I20260927 00:43:28.245877 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:28.245981 22401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:28.247526 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:28.249418 22401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:28.256244 22401 catalog_manager.cc:1347] Generated new cluster ID: 27e98ff5318b4f20bbb8ad20394e0cb9
I20260927 00:43:28.256332 22401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:28.261655 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:28.252814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37779 (local address 127.0.98.62:38585)
0927 00:43:28.252949 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:28.252954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:28.253248 (+ 294us) server_negotiation.cc:407] Connection header received
0927 00:43:28.253375 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:28.253378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:28.253434 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:28.253526 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:28.254423 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:28.254983 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:28.255853 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:28.256182 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:28.256321 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:28.258132 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:28.258136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:28.258139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:28.258173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:28.259865 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:28.260441 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:28.260445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:28.260447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:28.260550 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:28.260786 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:28.260788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:28.260790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:28.261018 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:28.261144 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:28.261348 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 00:43:28.261474 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:28.274660 22401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:28.275640 22401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:28.286105 22401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df304638e1be4a2ca9c84aa6592e8e8f: Generated new TSK 0
I20260927 00:43:28.286885 22401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:28.296115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22319
2026-09-27T00:43:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:28 dist-test-slave-xltq krb5kdc[22418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:28 dist-test-slave-xltq krb5kdc[22418](info): set up 2 sockets
Sep 27 00:43:28 dist-test-slave-xltq krb5kdc[22418](info): commencing operation
krb5kdc: starting...
W20260927 00:43:29.264282 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:29 dist-test-slave-xltq krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:29.383172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:29.490790 22431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:29.491061 22431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:29.491103 22431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:29.495085 22431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:29.495162 22431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:29.495180 22431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:29.495198 22431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:29.495211 22431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:29.500809 22431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:29.501873 22431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:29.502839 22431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:43:29.511816 22431 server_base.cc:1034] running on GCE node
W20260927 00:43:29.511894 22437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:29.511960 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:29.511894 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:29.512751 22431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:29.513986 22431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:29.515151 22431 hybrid_clock.cc:648] HybridClock initialized: now 1790469809515127 us; error 40 us; skew 500 ppm
Sep 27 00:43:29 dist-test-slave-xltq krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:29.517868 22431 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:29.518853 22431 webserver.cc:460] Webserver started at http://127.0.98.62:35213/ using document root <none> and password file <none>
I20260927 00:43:29.519394 22431 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:29.519464 22431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:29.519654 22431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:29.521827 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "905641bc910f4937bcb819175636011a"
format_stamp: "Formatted at 2026-09-27 00:43:29 on dist-test-slave-xltq"
server_key: "9c5237f7c70c3eb93691b1536244b956"
server_key_iv: "97e4f208f1b2987b1443c3460a29b811"
server_key_version: "encryptionkey@0"
I20260927 00:43:29.522459 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "905641bc910f4937bcb819175636011a"
format_stamp: "Formatted at 2026-09-27 00:43:29 on dist-test-slave-xltq"
server_key: "9c5237f7c70c3eb93691b1536244b956"
server_key_iv: "97e4f208f1b2987b1443c3460a29b811"
server_key_version: "encryptionkey@0"
I20260927 00:43:29.526608 22431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:29.529634 22446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:29.531222 22431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:29.531409 22431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "905641bc910f4937bcb819175636011a"
format_stamp: "Formatted at 2026-09-27 00:43:29 on dist-test-slave-xltq"
server_key: "9c5237f7c70c3eb93691b1536244b956"
server_key_iv: "97e4f208f1b2987b1443c3460a29b811"
server_key_version: "encryptionkey@0"
I20260927 00:43:29.531543 22431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:29.557173 22431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:29.560349 22431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:29.560529 22431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:29.567274 22431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41315
I20260927 00:43:29.567296 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41315 every 8 connection(s)
I20260927 00:43:29.568188 22431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:29.569885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22431
I20260927 00:43:29.570034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:29.570384 392 external_mini_cluster.cc:1428] Setting key b6781ddded2614931cbb9b79486e937c
I20260927 00:43:29.571355 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:29.577901 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: Bootstrap starting.
I20260927 00:43:29.579936 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:29.571925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50379 (local address 127.0.98.62:41315)
0927 00:43:29.572428 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:29.572443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:29.572487 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:43:29.573155 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:29.573172 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:29.573530 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:29.573861 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:29.574330 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:29.575346 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:29.576290 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:29.576833 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:29.577014 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:29.578062 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:29.578070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:29.578088 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:29.578545 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:29.578607 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:29.578633 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:29.578725 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:29.578867 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 00:43:29.579119 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":185,"threads_started":1}
I20260927 00:43:29.580418 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:29.581146 22499 log.cc:824] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:29.582839 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:29.583287 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: No bootstrap required, opened a new log
I20260927 00:43:29.587226 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 00:43:29.587492 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:29.587582 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905641bc910f4937bcb819175636011a, State: Initialized, Role: FOLLOWER
I20260927 00:43:29.588083 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 00:43:29.588294 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:29.588378 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:29.588493 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:29.589592 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 00:43:29.589994 22499 leader_election.cc:302] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 905641bc910f4937bcb819175636011a; no voters:
I20260927 00:43:29.590354 22499 leader_election.cc:288] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:29.590554 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:29.590888 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [term 1 LEADER]: Becoming Leader. State: Replica: 905641bc910f4937bcb819175636011a, State: Running, Role: LEADER
I20260927 00:43:29.591267 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 00:43:29.591708 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:29.592301 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:29.584888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39863 (local address 127.0.98.62:41315)
0927 00:43:29.585005 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:29.585009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:29.587858 (+ 2849us) server_negotiation.cc:407] Connection header received
0927 00:43:29.588003 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:29.588005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:29.588051 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:29.588109 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:29.588658 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:29.589230 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:29.590337 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:29.590663 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:29.590767 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:29.591742 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:29.591747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:29.591750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:29.591765 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:29.591798 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:29.591814 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:29.591896 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:29.592033 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 00:43:29.592110 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":49}
I20260927 00:43:29.593005 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "905641bc910f4937bcb819175636011a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } } }
I20260927 00:43:29.593149 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:29.593488 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 905641bc910f4937bcb819175636011a. Latest consensus state: current_term: 1 leader_uuid: "905641bc910f4937bcb819175636011a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "905641bc910f4937bcb819175636011a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } } }
I20260927 00:43:29.593629 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a [sys.catalog]: This master's current role is: LEADER
W20260927 00:43:29.596521 22524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:29.596606 22524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:29.596662 22518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:29.597361 22518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:29.602801 22518 catalog_manager.cc:1347] Generated new cluster ID: d7589abc9e764163a9dabb1fe60960c5
I20260927 00:43:29.602888 22518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:29.610610 22518 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:29.611689 22518 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:29.618642 22518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 905641bc910f4937bcb819175636011a: Generated new TSK 0
I20260927 00:43:29.619387 22518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:29.630064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22431
2026-09-27T00:43:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:29 dist-test-slave-xltq krb5kdc[22530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:29 dist-test-slave-xltq krb5kdc[22530](info): set up 2 sockets
Sep 27 00:43:29 dist-test-slave-xltq krb5kdc[22530](info): commencing operation
krb5kdc: starting...
W20260927 00:43:30.604449 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:30 dist-test-slave-xltq krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:30.727591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:30.841691 22543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:30.841997 22543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:30.842041 22543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:30.846325 22543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:30.846439 22543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:30.846460 22543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:30.846477 22543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:30.846491 22543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:30.852011 22543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:30.853166 22543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:30.854135 22543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:30.863263 22548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:30.863327 22543 server_base.cc:1034] running on GCE node
W20260927 00:43:30.863281 22551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:30.863256 22549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:30.864162 22543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:30.865459 22543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:30.866653 22543 hybrid_clock.cc:648] HybridClock initialized: now 1790469810866627 us; error 42 us; skew 500 ppm
Sep 27 00:43:30 dist-test-slave-xltq krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:30.869534 22543 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:30.870673 22543 webserver.cc:460] Webserver started at http://127.0.98.62:42167/ using document root <none> and password file <none>
I20260927 00:43:30.871363 22543 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:30.871428 22543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:30.871656 22543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:30.873850 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "3428101030264af5855ff5bc4a6ab2cc"
format_stamp: "Formatted at 2026-09-27 00:43:30 on dist-test-slave-xltq"
server_key: "94d05c2e10d04631c7bcc100355e6b0c"
server_key_iv: "ace48605c59d39f40170ce1ee71d7294"
server_key_version: "encryptionkey@0"
I20260927 00:43:30.874406 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "3428101030264af5855ff5bc4a6ab2cc"
format_stamp: "Formatted at 2026-09-27 00:43:30 on dist-test-slave-xltq"
server_key: "94d05c2e10d04631c7bcc100355e6b0c"
server_key_iv: "ace48605c59d39f40170ce1ee71d7294"
server_key_version: "encryptionkey@0"
I20260927 00:43:30.878291 22543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:43:30.881022 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:30.882403 22543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:30.882532 22543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "3428101030264af5855ff5bc4a6ab2cc"
format_stamp: "Formatted at 2026-09-27 00:43:30 on dist-test-slave-xltq"
server_key: "94d05c2e10d04631c7bcc100355e6b0c"
server_key_iv: "ace48605c59d39f40170ce1ee71d7294"
server_key_version: "encryptionkey@0"
I20260927 00:43:30.882653 22543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:30.887370 22543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:30.890542 22543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:30.890726 22543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:30.897969 22543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45031
I20260927 00:43:30.898069 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45031 every 8 connection(s)
I20260927 00:43:30.899004 22543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:30.902119 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:30.904093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22543
I20260927 00:43:30.904258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:30.904515 392 external_mini_cluster.cc:1428] Setting key befa76043afa6c1bed96eb2a1f744126
I20260927 00:43:30.909161 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc: Bootstrap starting.
I20260927 00:43:30.910013 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:30.905954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44271 (local address 127.0.98.62:45031)
0927 00:43:30.906494 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:30.906510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:30.906548 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:43:30.907312 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:30.907340 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:30.907648 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:30.907974 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:30.908785 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:30.908790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:30.908801 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:30.908811 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:30.908870 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:30.908891 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:30.909033 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:30.909102 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:43:30.909254 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":162,"threads_started":1}
I20260927 00:43:30.911511 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:30.912382 22611 log.cc:824] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:30.912878 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:30.914949 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc: No bootstrap required, opened a new log
I20260927 00:43:30.917263 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:30.915240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46385 (local address 127.0.98.62:45031)
0927 00:43:30.915381 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:30.915386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:30.915705 (+ 319us) server_negotiation.cc:407] Connection header received
0927 00:43:30.915860 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:30.915863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:30.915904 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:30.915988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:30.916797 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:30.916800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:30.916802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:43:30.916814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:30.916848 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:43:30.916860 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:30.916945 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:30.917024 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 00:43:30.917063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:43:30.919559 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } }
I20260927 00:43:30.919826 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:30.919876 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3428101030264af5855ff5bc4a6ab2cc, State: Initialized, Role: FOLLOWER
I20260927 00:43:30.920325 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } }
I20260927 00:43:30.920465 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:30.920531 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:30.920629 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:30.921579 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } }
I20260927 00:43:30.921922 22611 leader_election.cc:302] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3428101030264af5855ff5bc4a6ab2cc; no voters:
I20260927 00:43:30.922226 22611 leader_election.cc:288] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:30.922335 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:30.922573 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [term 1 LEADER]: Becoming Leader. State: Replica: 3428101030264af5855ff5bc4a6ab2cc, State: Running, Role: LEADER
I20260927 00:43:30.922881 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } }
I20260927 00:43:30.923444 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:30.924847 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3428101030264af5855ff5bc4a6ab2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } } }
I20260927 00:43:30.924878 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3428101030264af5855ff5bc4a6ab2cc. Latest consensus state: current_term: 1 leader_uuid: "3428101030264af5855ff5bc4a6ab2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3428101030264af5855ff5bc4a6ab2cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45031 } } }
I20260927 00:43:30.925046 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:30.925329 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:30.925402 22633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:30.928737 22633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:30.934906 22633 catalog_manager.cc:1347] Generated new cluster ID: 03be447227c54bbf90b83cdd3f4b76fe
I20260927 00:43:30.934998 22633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:30.956794 22633 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:30.957213 22633 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:30.970315 22633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3428101030264af5855ff5bc4a6ab2cc: Generated new TSK 0
I20260927 00:43:30.971236 22633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:30.978176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22543
2026-09-27T00:43:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1348 ms)
[----------] 18 tests from AuthTokenIssuingTest (24360 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:31 dist-test-slave-xltq krb5kdc[22642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:31 dist-test-slave-xltq krb5kdc[22642](info): set up 2 sockets
Sep 27 00:43:31 dist-test-slave-xltq krb5kdc[22642](info): commencing operation
krb5kdc: starting...
W20260927 00:43:31.948380 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:32.071287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:32.181502 22655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:32.181834 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:32.181859 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:32.181908 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:32.186123 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:32.186196 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:32.186214 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:32.186230 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:32.186245 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:32.191751 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:32.192879 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:32.193885 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:32.202593 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:32.202713 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:32.202764 22655 server_base.cc:1034] running on GCE node
W20260927 00:43:32.202502 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:32.203388 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:32.204523 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:32.205693 22655 hybrid_clock.cc:648] HybridClock initialized: now 1790469812205669 us; error 35 us; skew 500 ppm
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:32.208455 22655 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:32.209458 22655 webserver.cc:460] Webserver started at http://127.0.98.62:42641/ using document root <none> and password file <none>
I20260927 00:43:32.210069 22655 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:32.210135 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:32.210340 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:32.212512 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "c8385eca40104ce19bc6d5081249bfaa"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "7490914756d52ca6575c8d487ca47273"
server_key_iv: "698efa7875d21a3f2bc8be7754f33411"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.213128 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8385eca40104ce19bc6d5081249bfaa"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "7490914756d52ca6575c8d487ca47273"
server_key_iv: "698efa7875d21a3f2bc8be7754f33411"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.217147 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 00:43:32.220054 22670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:32.221484 22655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:43:32.221613 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "c8385eca40104ce19bc6d5081249bfaa"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "7490914756d52ca6575c8d487ca47273"
server_key_iv: "698efa7875d21a3f2bc8be7754f33411"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.221753 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:32.249439 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:32.252703 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:32.252879 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:32.259814 22655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260927 00:43:32.260092 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260927 00:43:32.260831 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:32.263725 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:32.267846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22655
I20260927 00:43:32.267946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:32.267788 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.268175 392 external_mini_cluster.cc:1428] Setting key 5ebabb6d7cff068c7d76a762568e5859
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:32.284422 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.270300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.0.1:39740)
0927 00:43:32.270858 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.270911 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.271963 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.272695 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.272700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.273186 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.273552 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.273573 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.274679 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.274683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.275836 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.275848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.276538 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.276541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.276645 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.277168 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.277184 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.279348 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.281488 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.281497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.281506 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.281717 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.282023 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.282026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.282028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.282131 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.282724 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.282731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.282857 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.283007 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.283395 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":87,"thread_start_us":143,"threads_started":1}
I20260927 00:43:32.284422 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.270300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.0.98.62:34329)
0927 00:43:32.270859 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.270876 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.271021 (+ 145us) server_negotiation.cc:407] Connection header received
0927 00:43:32.272316 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.272321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.272499 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.272647 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.273723 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.274554 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.275956 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.276396 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.276516 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.279499 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.279505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.279515 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.279537 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.281368 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.281815 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.281820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.281821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.281929 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.282208 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.282211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.282213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.282602 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.282728 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.283085 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.283309 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"spinlock_wait_cycles":77568,"thread_start_us":158,"threads_started":1}
I20260927 00:43:32.286866 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.287599 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.287344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35200)
0927 00:43:32.287458 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.287530 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:43:32.287762 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
I20260927 00:43:32.288611 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.288401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35214)
0927 00:43:32.288507 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.288552 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:43:32.288776 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46625: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:43:32.295650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46625
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:32.338049 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } attempt: 1
I20260927 00:43:32.338970 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.338656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35218)
0927 00:43:32.338820 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.338885 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 00:43:32.339811 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.339549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35222)
0927 00:43:32.339688 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.339743 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 00:43:32.339941 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46625: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
W20260927 00:43:32.406436 22730 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:32.406724 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:32.406747 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:32.406803 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:43:32.408090 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } attempt: 2
I20260927 00:43:32.408957 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.408670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35234)
0927 00:43:32.408818 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.408881 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 00:43:32.409796 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.409439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35240)
0927 00:43:32.409612 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.409717 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 00:43:32.409941 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46625: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46625: connect: Connection refused (error 111)
W20260927 00:43:32.411275 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:32.411332 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:32.411351 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:32.411366 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:32.411381 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:32.416872 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:32.417953 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:32.418957 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:32.428289 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:32.428273 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:32.428273 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:32.428319 22730 server_base.cc:1034] running on GCE node
I20260927 00:43:32.429276 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:32.430518 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:32.431715 22730 hybrid_clock.cc:648] HybridClock initialized: now 1790469812431690 us; error 40 us; skew 500 ppm
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:32.434953 22730 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:43:32.436004 22730 webserver.cc:460] Webserver started at http://127.0.98.61:42373/ using document root <none> and password file <none>
I20260927 00:43:32.436625 22730 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:32.436698 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:32.436908 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:32.439077 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data/instance:
uuid: "b6da7047146941049d3c43ef852cd0b0"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "8515dd18094186a134ae74ad04e2811e"
server_key_iv: "0d781c71923fc235b234d33b5a37534a"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.439685 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal/instance:
uuid: "b6da7047146941049d3c43ef852cd0b0"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "8515dd18094186a134ae74ad04e2811e"
server_key_iv: "0d781c71923fc235b234d33b5a37534a"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.443791 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:32.446574 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:32.447865 22730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:32.447985 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "b6da7047146941049d3c43ef852cd0b0"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "8515dd18094186a134ae74ad04e2811e"
server_key_iv: "0d781c71923fc235b234d33b5a37534a"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.448110 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:32.463649 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:32.466918 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:32.467118 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:32.474274 22730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46625
I20260927 00:43:32.474524 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46625 every 8 connection(s)
I20260927 00:43:32.475283 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:43:32.478158 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:32.482175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22730
I20260927 00:43:32.482285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-1/wal/instance
I20260927 00:43:32.482158 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.482511 392 external_mini_cluster.cc:1428] Setting key af3ff732236bac8b1e845e872ec8ab34
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:32.497447 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.484250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.0.98.62:34329)
0927 00:43:32.484410 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.484415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.485332 (+ 917us) server_negotiation.cc:407] Connection header received
0927 00:43:32.486350 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.486354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.486421 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.486538 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.487748 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.488623 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.489849 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.490199 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.490297 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.493571 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.493574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.493576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.493606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.495041 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.495605 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.495609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.495610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.495703 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.496076 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.496079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.496081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.496380 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.496493 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.497109 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.497262 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:43:32.498293 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.484633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.0.1:39746)
0927 00:43:32.485147 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.485207 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.486126 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.486719 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.486727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.487079 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.487583 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.487596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.488785 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.488788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.489700 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.489720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.490390 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.490394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.490567 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.491229 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.491246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.493340 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.495198 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.495205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.495221 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.495499 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.495792 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.495796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.495798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.495951 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.496503 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.496509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.496826 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.497007 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.497391 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260927 00:43:32.499954 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.501124 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } attempt: 3
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:43:32.510619 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.500532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35250)
0927 00:43:32.500644 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.500656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.500768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.501589 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.501592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.501858 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.501987 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.501994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.503240 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.503243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.503849 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.503858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.504787 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.504790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.504897 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.505511 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.505522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.506839 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.508864 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.508868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.508870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.509133 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.509460 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.509464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.509466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.509558 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.510099 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.510102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.510189 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.510284 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.510428 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:43:32.510751 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.500695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.98.61:46625)
0927 00:43:32.501003 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.501012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.501032 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:43:32.501093 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.501097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.501284 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.501420 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.502103 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.503064 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.503959 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.504580 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.504688 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.506987 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.507000 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.507021 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.507046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.508758 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.509280 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.509287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.509289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.509372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.509648 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.509651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.509652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.510005 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.510133 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.510378 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.510510 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260927 00:43:32.510900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32823
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:32.512818 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.501653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.0.98.61:46625)
0927 00:43:32.502019 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.502027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.502038 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:43:32.502068 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.502071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.502109 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.502193 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.502677 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.503339 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.504767 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.505100 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.505197 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.507496 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.507498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.507500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.507521 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.509563 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.510038 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.510042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.510043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.510137 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.510487 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.510489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.510491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.512066 (+ 1575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.512194 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.512555 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.512672 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":153,"threads_started":1}
I20260927 00:43:32.512840 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.501564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35264)
0927 00:43:32.501698 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.501733 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.501819 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.502241 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.502244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.502419 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.502544 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.502551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.503458 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.503461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.504639 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.504653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.505272 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.505276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.505390 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.505758 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.505771 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.507339 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.509690 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.509693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.509696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.509940 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.510271 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.510275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.510277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.510374 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.512194 (+ 1820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.512198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.512356 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.512502 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.512665 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 00:43:32.513306 22798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.514145 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.513836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43392)
0927 00:43:32.513963 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.514064 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 00:43:32.514351 22747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.514737 22723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.515214 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.515023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43398)
0927 00:43:32.515111 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.515159 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 00:43:32.515436 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.515195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43402)
0927 00:43:32.515326 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.515374 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 00:43:32.515594 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.516113 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.515887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43416)
0927 00:43:32.516017 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.516061 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:43:32.516306 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.547488 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 1
I20260927 00:43:32.548511 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.548110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43426)
0927 00:43:32.548346 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.548421 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 00:43:32.549314 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.549044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43430)
0927 00:43:32.549152 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.549201 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:43:32.549439 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.564895 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 1
I20260927 00:43:32.565793 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.565479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43442)
0927 00:43:32.565649 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.565712 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 00:43:32.566618 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.566366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43454)
0927 00:43:32.566480 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.566547 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:43:32.566782 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.624617 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 2
I20260927 00:43:32.625453 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.625115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43460)
0927 00:43:32.625316 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.625373 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 00:43:32.626139 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.625905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43472)
0927 00:43:32.626027 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.626075 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:43:32.626262 22723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
W20260927 00:43:32.626945 22805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:32.627261 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:32.627301 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:32.627358 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:32.631675 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:32.631750 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:32.631778 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:32.631801 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:32.631834 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:32.634965 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 2
I20260927 00:43:32.635857 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.635520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43476)
0927 00:43:32.635708 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.635774 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 00:43:32.636669 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.636452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43480)
0927 00:43:32.636554 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.636611 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:43:32.636782 22798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32823: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32823: connect: Connection refused (error 111)
I20260927 00:43:32.637813 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329,127.0.98.61:46625,127.0.98.60:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:32.638928 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:32.639936 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:32.648629 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:32.648623 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:32.648669 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:32.648940 22805 server_base.cc:1034] running on GCE node
I20260927 00:43:32.649510 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:32.650697 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:32.651873 22805 hybrid_clock.cc:648] HybridClock initialized: now 1790469812651836 us; error 53 us; skew 500 ppm
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:32.654722 22805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:43:32.655697 22805 webserver.cc:460] Webserver started at http://127.0.98.60:36773/ using document root <none> and password file <none>
I20260927 00:43:32.656280 22805 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:32.656353 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:32.656533 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:32.658601 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data/instance:
uuid: "971fde0ba6074028bd93a96a3e4d0254"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "e4d0ccb20e4af4abff9cccddd74b97b9"
server_key_iv: "f854e591df72bdade404b1cfa05114f4"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.659184 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal/instance:
uuid: "971fde0ba6074028bd93a96a3e4d0254"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "e4d0ccb20e4af4abff9cccddd74b97b9"
server_key_iv: "f854e591df72bdade404b1cfa05114f4"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.663372 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:43:32.666111 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:32.667474 22805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:32.667593 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "971fde0ba6074028bd93a96a3e4d0254"
format_stamp: "Formatted at 2026-09-27 00:43:32 on dist-test-slave-xltq"
server_key: "e4d0ccb20e4af4abff9cccddd74b97b9"
server_key_iv: "f854e591df72bdade404b1cfa05114f4"
server_key_version: "encryptionkey@0"
I20260927 00:43:32.667721 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:32.686154 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:32.689595 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:32.689792 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:32.696909 22805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32823
I20260927 00:43:32.697201 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32823 every 8 connection(s)
I20260927 00:43:32.697925 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:43:32.699970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 22805
I20260927 00:43:32.700089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469528995939-392-0/minicluster-data/master-2/wal/instance
I20260927 00:43:32.700413 392 external_mini_cluster.cc:1428] Setting key cefae6982460de81d5b6e6f7fd61bd93
I20260927 00:43:32.701419 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:32.706002 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:32.716358 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.701860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.98.60:32823)
0927 00:43:32.702393 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.702412 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.702449 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:32.703184 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.703210 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.703579 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.703899 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.704925 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.706073 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.707357 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.707982 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.708178 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.711287 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.711296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.711306 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.711342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.713111 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.713563 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.713569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.713576 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.713738 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.714004 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.714007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.714009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.714495 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.714702 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.715082 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.715495 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":144,"threads_started":1}
I20260927 00:43:32.717360 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.708129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.0.1:39750)
0927 00:43:32.708595 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.708646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.708775 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.709076 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.709162 (+ 86us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.709532 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.709699 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.709705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.710718 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.710722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.711438 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.711448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.711902 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.711905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.712025 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.712540 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.712553 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.713957 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.715789 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.715798 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.715805 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.716024 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.716333 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.716336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.716339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.716414 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.716847 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.716852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.716957 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.717079 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.717228 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":206,"threads_started":1}
I20260927 00:43:32.717669 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.708185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.0.98.62:34329)
0927 00:43:32.708359 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.708364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.708743 (+ 379us) server_negotiation.cc:407] Connection header received
0927 00:43:32.708870 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.708873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.708931 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.709005 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.709837 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.710600 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.711566 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.711796 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.711888 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.714114 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.714117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.714119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.714147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.715673 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.716123 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.716126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.716127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.716234 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.716494 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.716497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.716499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.716744 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.716842 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.717162 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.717294 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 00:43:32.718536 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:32.725411 22723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 3
I20260927 00:43:32.727485 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.719262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.0.98.61:46625)
0927 00:43:32.719386 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.719390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.719403 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:32.719443 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.719445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.719491 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.719570 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.719967 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.720616 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.721507 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.721822 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.721920 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.724004 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.724007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.724009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.724040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.725515 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.726108 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.726112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.726114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.726204 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.726507 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.726511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.726513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.726705 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.726789 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.727098 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.727319 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:43:32.727937 22798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } attempt: 3
I20260927 00:43:32.728018 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.719018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46625 (local address 127.0.0.1:35268)
0927 00:43:32.719127 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.719140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.719204 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.719598 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.719601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.719743 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.719854 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.719860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.720733 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.720736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.721385 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.721395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.721936 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.721938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.722010 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.722531 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.722542 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.723841 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.725656 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.725660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.725662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.725948 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.726309 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.726311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.726313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.726403 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.726827 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.726830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.726922 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.727632 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.727776 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 00:43:32.728950 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:32.729413 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.718895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.0.98.62:34329)
0927 00:43:32.719006 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.719010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.719047 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:32.719194 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.719197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.719244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.719327 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.720143 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.720721 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.721495 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.721710 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.721788 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.724713 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.724717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.724719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.724748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.726494 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.726976 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.726979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.726981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.727062 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.727310 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.727313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.727315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.727486 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.727725 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.729132 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.729267 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:32.736524 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.725942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.98.60:32823)
0927 00:43:32.726112 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.726116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.726128 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:43:32.726267 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.726269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.726315 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.726387 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.726798 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.727527 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.728578 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.728835 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.728936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.731922 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.731926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.731928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.731953 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.734190 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.735172 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.735176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.735178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.735275 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.735615 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.735618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.735620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.735830 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.735922 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.736230 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.736352 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 00:43:32.736545 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.725868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43504)
0927 00:43:32.726080 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.726093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.726174 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.726403 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.726405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.726576 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.726684 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.726691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.727654 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.727658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.728451 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.728463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.729057 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.729061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.729141 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.729853 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.729921 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.731775 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.734318 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.734322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.734324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.734582 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.735385 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.735389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.735391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.735504 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.735941 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.735944 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.736043 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.736136 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.736395 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
Sep 27 00:43:32 dist-test-slave-xltq krb5kdc[22642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:43:32.740923 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.729463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43522)
0927 00:43:32.729808 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.729822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.729894 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.731065 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.731068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.731258 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.731354 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.731361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.732408 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.732412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.732977 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.732986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.734016 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.734020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.734105 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.734688 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.734702 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.736930 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.739424 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.739427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.739429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.739658 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.739923 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.739926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.739927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.740003 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.740450 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.740453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.740531 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.740604 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.740751 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260927 00:43:32.740978 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa: Bootstrap starting.
I20260927 00:43:32.741299 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.728374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32823 (local address 127.0.0.1:43510)
0927 00:43:32.728594 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:32.728607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:32.728674 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:32.730380 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:32.730383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:32.730557 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:32.730778 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.730787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.732579 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.732583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.733347 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:32.733359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.734017 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.734020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:32.734121 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.734599 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:32.734613 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:32.736672 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:32.739482 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.739485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.739488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.739837 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.740247 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:32.740250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:32.740252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:32.740343 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:32.740764 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:32.740767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:32.740883 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 00:43:32.740978 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:43:32.741130 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":3968}
I20260927 00:43:32.742072 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.728812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.0.98.60:32823)
0927 00:43:32.730131 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.730140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.730151 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:43:32.730181 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.730184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.730224 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.730310 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.730929 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.732164 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.733445 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.733883 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.734321 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.737030 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.737034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.737036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.737074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.739351 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.739948 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.739952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.739953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.740108 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.740455 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.740458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.740460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.740665 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.741609 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.741690 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.741847 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":1256,"thread_start_us":155,"threads_started":1}
I20260927 00:43:32.742168 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.730724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.0.98.61:46625)
0927 00:43:32.730835 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.730839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.731353 (+ 514us) server_negotiation.cc:407] Connection header received
0927 00:43:32.731479 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.731483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.731527 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.731590 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.732411 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.733170 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.734857 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.735175 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.735272 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.738168 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.738172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.738175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.738201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.739877 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.740443 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.740446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.740448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.740524 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.740841 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.740844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.740846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.741021 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.741227 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.741921 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.742037 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:43:32.742105 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.729703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.0.98.60:32823)
0927 00:43:32.730841 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.730850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.730864 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:43:32.730895 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.730898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.730941 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.731027 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.731469 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.732287 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.733454 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.733891 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.734315 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.737075 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.737078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.737081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.737104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.739311 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.739763 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.739766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.739768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.739842 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.740089 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.740091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.740093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.740355 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.741626 (+ 1271us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.741700 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.741920 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":1075,"spinlock_wait_cycles":54656,"thread_start_us":94,"threads_started":1}
I20260927 00:43:32.744999 22723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:32.745522 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: Bootstrap starting.
I20260927 00:43:32.745985 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: Bootstrap starting.
I20260927 00:43:32.745968 22723 log.cc:824] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:32.747042 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:32.748512 22798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:32.749114 22874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:32.748363 22697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8385eca40104ce19bc6d5081249bfaa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39752
I20260927 00:43:32.749403 22798 log.cc:824] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:32.749887 22874 log.cc:824] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:32.751621 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: No bootstrap required, opened a new log
I20260927 00:43:32.751634 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: No bootstrap required, opened a new log
I20260927 00:43:32.753645 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa: No bootstrap required, opened a new log
I20260927 00:43:32.755359 22874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.755443 22798 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.755599 22874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971fde0ba6074028bd93a96a3e4d0254, State: Initialized, Role: FOLLOWER
I20260927 00:43:32.755699 22798 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6da7047146941049d3c43ef852cd0b0, State: Initialized, Role: FOLLOWER
I20260927 00:43:32.755915 22874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.755995 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.756397 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.756596 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.756632 22723 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.756840 22723 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8385eca40104ce19bc6d5081249bfaa, State: Initialized, Role: FOLLOWER
I20260927 00:43:32.757120 22723 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.757232 22874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:32.757292 22798 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:32.757625 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.757818 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.758423 22723 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:32.760293 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:43:32.760414 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:32.761742 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
W20260927 00:43:32.763039 22911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:32.763134 22911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 00:43:32.763101 22913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:43:32.763206 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8385eca40104ce19bc6d5081249bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6da7047146941049d3c43ef852cd0b0"
W20260927 00:43:32.763321 22913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:32.763443 22772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:32.761744 22730 master_runner.cc:186] Error getting master registration for 127.0.98.62:34329: OK, instance_id { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" instance_seqno: 1790469812257864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:43:32.764806 22772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8385eca40104ce19bc6d5081249bfaa in term 1.
I20260927 00:43:32.764948 22730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:32.765244 22674 leader_election.cc:302] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6da7047146941049d3c43ef852cd0b0, c8385eca40104ce19bc6d5081249bfaa; no voters:
I20260927 00:43:32.765558 22886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:32.765596 22697 leader_election.cc:288] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [CANDIDATE]: Term 1 election: Requested vote from peers b6da7047146941049d3c43ef852cd0b0 (127.0.98.61:46625), 971fde0ba6074028bd93a96a3e4d0254 (127.0.98.60:32823)
I20260927 00:43:32.765797 22886 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [term 1 LEADER]: Becoming Leader. State: Replica: c8385eca40104ce19bc6d5081249bfaa, State: Running, Role: LEADER
I20260927 00:43:32.765816 22848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8385eca40104ce19bc6d5081249bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "971fde0ba6074028bd93a96a3e4d0254"
I20260927 00:43:32.766043 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:32.766149 22886 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } }
I20260927 00:43:32.764285 22805 master_runner.cc:186] Error getting master registration for 127.0.98.62:34329: OK, instance_id { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" instance_seqno: 1790469812257864 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:43:32.767405 22848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8385eca40104ce19bc6d5081249bfaa in term 1.
I20260927 00:43:32.767490 22805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:32.768024 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.768172 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.768131 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8385eca40104ce19bc6d5081249bfaa. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.768451 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:32.768822 22927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:32.772461 22927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:32.783972 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.772611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.0.98.62:34329)
0927 00:43:32.772727 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.772731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.774602 (+ 1871us) server_negotiation.cc:407] Connection header received
0927 00:43:32.774745 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.774749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.774798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.774870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.775891 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.776655 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.777805 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.778133 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.778209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.780143 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.780148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.780150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.780184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.781952 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.782477 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.782482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.782484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.782596 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.782932 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.782936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.782937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.783173 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.783280 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.783656 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.783793 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:43:32.788765 22848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [term 1 FOLLOWER]: Refusing update from remote peer c8385eca40104ce19bc6d5081249bfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:32.789666 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.773345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.0.98.61:46625)
0927 00:43:32.773449 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.773453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.774916 (+ 1463us) server_negotiation.cc:407] Connection header received
0927 00:43:32.775460 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.775463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.775515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.775566 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.778493 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.779261 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.781439 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.781749 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.781840 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.785949 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.785953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.785956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.785991 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.787649 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.788191 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.788195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.788197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.788328 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.788654 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.788657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.788659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.788899 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.789012 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.789358 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.789496 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:43:32.789758 22921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:32.795470 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8385eca40104ce19bc6d5081249bfaa. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.795641 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.797055 22772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [term 1 FOLLOWER]: Refusing update from remote peer c8385eca40104ce19bc6d5081249bfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:32.798002 22921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:32.799526 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.799667 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:32.801184 22933 mvcc.cc:205] Tried to move back new op lower bound from 7333764353172729856 to 7333764353093812224. Current Snapshot: MvccSnapshot[applied={T|T < 7333764353172729856}]
I20260927 00:43:32.801388 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:32.773818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.0.98.60:32823)
0927 00:43:32.773936 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.773941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.775117 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 00:43:32.776547 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.776551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.776602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.776651 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.777571 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.778352 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.782530 (+ 4178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:32.782940 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:32.783026 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:32.796378 (+ 13352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:32.796383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:32.796386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:32.796424 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:32.798531 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.799184 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.799188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.799190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.799293 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:32.799865 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:32.799869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:32.799871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:32.800134 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:32.800317 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:32.801078 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 00:43:32.801222 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:43:32.801752 22934 mvcc.cc:205] Tried to move back new op lower bound from 7333764353172729856 to 7333764353093812224. Current Snapshot: MvccSnapshot[applied={T|T < 7333764353172729856}]
I20260927 00:43:32.802708 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8385eca40104ce19bc6d5081249bfaa. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.802831 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.802940 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.803013 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.803490 22927 catalog_manager.cc:1347] Generated new cluster ID: c06f2c9b87ba4d73b37a312f61a600af
I20260927 00:43:32.803553 22927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:32.816870 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.817027 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:32.818382 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8385eca40104ce19bc6d5081249bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8385eca40104ce19bc6d5081249bfaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } peers { permanent_uuid: "b6da7047146941049d3c43ef852cd0b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46625 } } peers { permanent_uuid: "971fde0ba6074028bd93a96a3e4d0254" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32823 } } }
I20260927 00:43:32.818516 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:32.819413 22938 mvcc.cc:205] Tried to move back new op lower bound from 7333764353172729856 to 7333764353093812224. Current Snapshot: MvccSnapshot[applied={T|T < 7333764353172729856}]
I20260927 00:43:32.856299 22927 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:32.857890 22927 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:32.884380 22927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8385eca40104ce19bc6d5081249bfaa: Generated new TSK 0
I20260927 00:43:32.885345 22927 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:43:32.922979 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.919712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.0.98.60:32823)
0927 00:43:32.919843 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.919847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.921828 (+ 1981us) server_negotiation.cc:407] Connection header received
0927 00:43:32.922035 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.922039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.922092 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.922157 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.922820 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43532: BlockingRecv error: recv got EOF from 127.0.0.1:43532 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:43:32.924750 22727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.920243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.0.98.62:34329)
0927 00:43:32.920357 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.920361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.923719 (+ 3358us) server_negotiation.cc:407] Connection header received
0927 00:43:32.923903 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.923906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.923959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.924010 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.924610 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39766: BlockingRecv error: recv got EOF from 127.0.0.1:39766 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:43:32.926453 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.921209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.0.98.61:46625)
0927 00:43:32.921326 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.921330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.925424 (+ 4094us) server_negotiation.cc:407] Connection header received
0927 00:43:32.925599 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.925602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.925656 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.925709 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.926324 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35292: BlockingRecv error: recv got EOF from 127.0.0.1:35292 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:43:32.931694 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.929952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.0.98.61:46625)
0927 00:43:32.930069 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.930074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.930593 (+ 519us) server_negotiation.cc:407] Connection header received
0927 00:43:32.930837 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.930840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.930920 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.930976 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.931568 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35298: BlockingRecv error: recv got EOF from 127.0.0.1:35298 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:43:32.940423 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.938935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.98.61:46625)
0927 00:43:32.939077 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.939081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.939345 (+ 264us) server_negotiation.cc:407] Connection header received
0927 00:43:32.939550 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.939554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.939609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.939661 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.940292 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35314: BlockingRecv error: recv got EOF from 127.0.0.1:35314 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 00:43:32.949620 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.946024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.98.61:46625)
0927 00:43:32.946140 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.946144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.948552 (+ 2408us) server_negotiation.cc:407] Connection header received
0927 00:43:32.948779 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.948784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.948849 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.948904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.949494 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35326: BlockingRecv error: recv got EOF from 127.0.0.1:35326 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 00:43:32.958623 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.956443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.98.61:46625)
0927 00:43:32.956602 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.956607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.957318 (+ 711us) server_negotiation.cc:407] Connection header received
0927 00:43:32.957620 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.957624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.957686 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.957741 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.958479 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35338: BlockingRecv error: recv got EOF from 127.0.0.1:35338 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 00:43:32.972150 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.970298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.98.61:46625)
0927 00:43:32.970423 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.970428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.971045 (+ 617us) server_negotiation.cc:407] Connection header received
0927 00:43:32.971302 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.971306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.971364 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.971420 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.972029 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:43:32.992473 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:32.991145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.0.98.61:46625)
0927 00:43:32.991278 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:32.991282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:32.991502 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:43:32.991660 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:32.991663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:32.991721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:32.991817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:32.992340 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35364: BlockingRecv error: recv got EOF from 127.0.0.1:35364 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:43:33.027807 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:33.026672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.0.98.61:46625)
0927 00:43:33.026794 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:33.026798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:33.026860 (+ 62us) server_negotiation.cc:407] Connection header received
0927 00:43:33.026988 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:33.026990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:33.027037 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:33.027209 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:33.027706 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35376: BlockingRecv error: recv got EOF from 127.0.0.1:35376 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:43:33.095698 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:33.094424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.0.98.61:46625)
0927 00:43:33.094595 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:33.094600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:33.094699 (+ 99us) server_negotiation.cc:407] Connection header received
0927 00:43:33.094908 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:33.094911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:33.094960 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:33.095041 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:33.095576 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35392: BlockingRecv error: recv got EOF from 127.0.0.1:35392 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 00:43:33.226872 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:33.225789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.98.61:46625)
0927 00:43:33.225944 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:33.225949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:33.226107 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:43:33.226230 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:33.226232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:33.226293 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:33.226379 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:33.226743 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35398: BlockingRecv error: recv got EOF from 127.0.0.1:35398 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:43:33.486121 22804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:33.484996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.0.98.61:46625)
0927 00:43:33.485123 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:33.485127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:33.485214 (+ 87us) server_negotiation.cc:407] Connection header received
0927 00:43:33.485349 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:33.485352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:33.485401 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:33.485471 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:33.486007 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35402: BlockingRecv error: recv got EOF from 127.0.0.1:35402 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:43:33.764747 22913 catalog_manager.cc:1259] Loaded cluster ID: c06f2c9b87ba4d73b37a312f61a600af
I20260927 00:43:33.764747 22911 catalog_manager.cc:1259] Loaded cluster ID: c06f2c9b87ba4d73b37a312f61a600af
I20260927 00:43:33.764844 22913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: loading cluster ID for follower catalog manager: success
I20260927 00:43:33.764844 22911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: loading cluster ID for follower catalog manager: success
I20260927 00:43:33.767115 22913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: acquiring CA information for follower catalog manager: success
I20260927 00:43:33.767115 22911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: acquiring CA information for follower catalog manager: success
I20260927 00:43:33.768671 22911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 971fde0ba6074028bd93a96a3e4d0254: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:33.768671 22913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6da7047146941049d3c43ef852cd0b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:34.004426 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:34.000331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.98.61:46625)
0927 00:43:34.000562 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:34.000566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:34.000628 (+ 62us) server_negotiation.cc:407] Connection header received
0927 00:43:34.000805 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:34.000808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:34.000990 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:34.001098 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:34.001352 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.001915 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.002783 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.003069 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.003154 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:34.003491 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:34.003982 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:34.004076 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:34.004170 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:43:34.004261 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260927 00:43:34.012231 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:34.007691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.98.60:32823)
0927 00:43:34.007921 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:34.007925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:34.007939 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:43:34.008133 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:34.008135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:34.008294 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:34.008397 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:34.008768 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.009322 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.010347 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.010666 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.010756 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:34.011179 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:34.011730 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:34.011839 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:34.011978 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:43:34.012040 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260927 00:43:34.019695 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:34.015768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.0.98.60:32823)
0927 00:43:34.015968 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:34.015973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:34.016046 (+ 73us) server_negotiation.cc:407] Connection header received
0927 00:43:34.016241 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:34.016244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:34.016301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:34.016352 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:34.016691 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.017441 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.018566 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.018830 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.018933 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:34.019212 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:34.019308 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:34.019387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:34.019479 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 00:43:34.019544 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 00:43:34.020089 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:34.015721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.62:34329)
0927 00:43:34.015985 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:34.015989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:34.016335 (+ 346us) server_negotiation.cc:407] Connection header received
0927 00:43:34.016455 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:34.016458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:34.016625 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:34.016751 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:34.017000 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.017631 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.018617 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.018967 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.019064 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:34.019340 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:34.019737 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:34.019818 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:34.019890 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:43:34.019951 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260927 00:43:34.020305 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:34.016073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.0.98.61:46625)
0927 00:43:34.016243 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:34.016247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:34.016557 (+ 310us) server_negotiation.cc:407] Connection header received
0927 00:43:34.016640 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:34.016643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:34.016696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:34.016774 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:34.017135 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.017877 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.019063 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:34.019366 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:34.019438 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:34.019793 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:34.019884 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:34.019935 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:34.020092 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:43:34.020151 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:43:34.031309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22655
I20260927 00:43:34.037289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22730
I20260927 00:43:34.042634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 22805
2026-09-27T00:43:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3066 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:34 dist-test-slave-xltq krb5kdc[23022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:34 dist-test-slave-xltq krb5kdc[23022](info): set up 2 sockets
Sep 27 00:43:34 dist-test-slave-xltq krb5kdc[23022](info): commencing operation
krb5kdc: starting...
W20260927 00:43:35.008473 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:35.133002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:35.247715 23035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:35.248050 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:35.248073 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:35.248112 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:35.252429 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:35.252502 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:35.252521 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:35.252537 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:35.252560 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:35.258246 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:35.259424 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:35.260460 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:35.269074 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:35.269286 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:35.269362 23035 server_base.cc:1034] running on GCE node
W20260927 00:43:35.269263 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:35.270099 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:35.271344 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:35.272516 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790469815272503 us; error 30 us; skew 500 ppm
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:35.275364 23035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:35.276319 23035 webserver.cc:460] Webserver started at http://127.0.98.62:35561/ using document root <none> and password file <none>
I20260927 00:43:35.276859 23035 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:35.276919 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:35.277091 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:35.279161 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data/instance:
uuid: "a0f11b3f51134310adde3903206196da"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "eb67d0cfae9d999f936f70ad9775bb2d"
server_key_iv: "6247f797b4a5a13460826603bf6aa7ba"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.279698 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0f11b3f51134310adde3903206196da"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "eb67d0cfae9d999f936f70ad9775bb2d"
server_key_iv: "6247f797b4a5a13460826603bf6aa7ba"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.283836 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:43:35.286481 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:35.287730 23035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:43:35.287868 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal
uuid: "a0f11b3f51134310adde3903206196da"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "eb67d0cfae9d999f936f70ad9775bb2d"
server_key_iv: "6247f797b4a5a13460826603bf6aa7ba"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.287989 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:35.304518 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:35.307834 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:35.308010 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:35.315316 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41147
I20260927 00:43:35.315594 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41147 every 8 connection(s)
I20260927 00:43:35.316388 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:35.319769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 23035
I20260927 00:43:35.319818 23103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:35.319903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:35.320182 392 external_mini_cluster.cc:1428] Setting key c14dfae584b7b3b5b9455a87bd5f9107
I20260927 00:43:35.324286 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:35.341981 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.326910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41147 (local address 127.0.0.1:45944)
0927 00:43:35.327547 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.327596 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.328648 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.329487 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.329493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.329860 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.330182 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.330197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.331684 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.331690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.332720 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.332735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.333514 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.333519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.333709 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.334328 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.334348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.336533 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.338773 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.338783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.338795 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.339117 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.339448 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.339451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.339452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.339572 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.340449 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.340454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.340592 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.340737 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.341088 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":1,"thread_start_us":188,"threads_started":1}
I20260927 00:43:35.341981 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.328034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.0.98.62:41147)
0927 00:43:35.328350 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.328360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.328397 (+ 37us) server_negotiation.cc:407] Connection header received
0927 00:43:35.328996 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.329000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.329250 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.329428 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.330388 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.331510 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.332867 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.333333 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.333473 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.336757 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.336764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.336778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.336805 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.338621 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.339244 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.339250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.339251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.339364 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.339773 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.339776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.339778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.340321 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.340440 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.340920 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.341160 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260927 00:43:35.344655 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.345535 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.345243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43572)
0927 00:43:35.345375 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.345459 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:43:35.345762 23052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
I20260927 00:43:35.346714 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.346501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43588)
0927 00:43:35.346600 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.346651 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab.
W20260927 00:43:35.347025 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34403: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
I20260927 00:43:35.347858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34403
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:35.396313 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } attempt: 1
I20260927 00:43:35.397298 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.396936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43604)
0927 00:43:35.397110 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.397177 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 00:43:35.398118 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.397867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43620)
0927 00:43:35.398005 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.398052 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 00:43:35.398247 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34403: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
W20260927 00:43:35.462615 23110 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:35.462950 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:35.462985 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:35.463048 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:43:35.466413 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } attempt: 2
I20260927 00:43:35.467236 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.466929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43624)
0927 00:43:35.467090 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.467155 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 00:43:35.467633 23110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:35.467710 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:35.467748 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:35.467779 23110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:35.467808 23110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:35.468117 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.467865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43640)
0927 00:43:35.467973 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.468028 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:43:35.468273 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34403: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34403: connect: Connection refused (error 111)
I20260927 00:43:35.473775 23110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:35.475008 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:35.476027 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:35.484696 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:35.484802 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:35.484719 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:35.485023 23110 server_base.cc:1034] running on GCE node
I20260927 00:43:35.485610 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:35.486809 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:35.487986 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790469815487971 us; error 37 us; skew 500 ppm
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:35.490836 23110 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:43:35.491830 23110 webserver.cc:460] Webserver started at http://127.0.98.61:44611/ using document root <none> and password file <none>
I20260927 00:43:35.492447 23110 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:35.492511 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:35.492720 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:35.494846 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data/instance:
uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "ecfbaacdf145b30fbc5cf9b60ad9c79c"
server_key_iv: "38cad1247dcee849e003de73398e7378"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.495379 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal/instance:
uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "ecfbaacdf145b30fbc5cf9b60ad9c79c"
server_key_iv: "38cad1247dcee849e003de73398e7378"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.499632 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:43:35.502485 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:35.503796 23110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 00:43:35.503934 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal
uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "ecfbaacdf145b30fbc5cf9b60ad9c79c"
server_key_iv: "38cad1247dcee849e003de73398e7378"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.504055 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:35.515445 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:35.519115 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:35.519344 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:35.527222 23110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34403
I20260927 00:43:35.527511 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34403 every 8 connection(s)
I20260927 00:43:35.528450 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:43:35.531440 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:35.534981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 23110
I20260927 00:43:35.535116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-1/wal/instance
I20260927 00:43:35.535414 392 external_mini_cluster.cc:1428] Setting key c6d180e7db6f99259676d39c20f3edb6
I20260927 00:43:35.535543 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:35.551146 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.537657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.62:41147)
0927 00:43:35.537830 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.537836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.538740 (+ 904us) server_negotiation.cc:407] Connection header received
0927 00:43:35.539744 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.539748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.539821 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.539930 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.541361 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.542182 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.543721 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.544064 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.544172 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.547392 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.547401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.547404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.547433 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.548857 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.549445 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.549449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.549450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.549535 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.549903 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.549906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.549908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.550111 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.550203 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.550826 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.550973 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:43:35.552131 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.538019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41147 (local address 127.0.0.1:45954)
0927 00:43:35.538552 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.538612 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.539523 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.540129 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.540137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.540576 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.541169 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.541184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.542372 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.542375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.543540 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.543557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.544290 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.544293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.544463 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.545170 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.545190 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.547168 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.549002 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.549010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.549032 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.549334 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.549624 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.549627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.549629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.549804 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.550220 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.550228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.550540 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.550746 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.551131 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260927 00:43:35.553915 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.559504 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } attempt: 3
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:43:35.563709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
/tmp/dist-test-taskSbnIrP/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32909
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:43:35.565794 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.554515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43644)
0927 00:43:35.554706 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.554722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.554896 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.555417 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.555420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.555570 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.555683 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.555689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.556808 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.556811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.557376 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.557384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.558089 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.558093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.558188 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.558636 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.558648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.560053 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.562331 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.562335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.562337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.562547 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.562853 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.562856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.562858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.562955 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.565272 (+ 2317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.565275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.565363 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.565455 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.565598 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 00:43:35.565834 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.554631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.98.61:34403)
0927 00:43:35.555000 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.555009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.555028 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:43:35.555065 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.555069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.555279 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.555441 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.555844 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.556706 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.557506 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.557967 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.558061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.560232 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.560244 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.560312 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.560347 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.562198 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.562650 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.562657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.562659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.562766 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.563055 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.563058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.563060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.565154 (+ 2094us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.565308 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.565549 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.565680 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260927 00:43:35.568336 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.569133 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.568828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37848)
0927 00:43:35.568962 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.569045 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 00:43:35.569514 23129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.570191 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.559991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43646)
0927 00:43:35.560128 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.560143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.560243 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.560599 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.560602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.560747 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.560954 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.560962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.561869 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.561872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.562593 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.562605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.563118 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.563122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.563241 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.563648 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.563660 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.564982 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.568583 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.568586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.568589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.568821 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.569145 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.569148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.569150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.569273 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.569653 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.569657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.569753 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.569874 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.570032 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 00:43:35.570554 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.570288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37860)
0927 00:43:35.570433 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.570483 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 00:43:35.570715 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.560056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.0.98.61:34403)
0927 00:43:35.560373 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.560382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.560395 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:35.560428 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.560431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.560471 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.560539 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.561083 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.561738 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.562698 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.562984 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.563107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.565162 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.565166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.565168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.565208 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.568420 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.568943 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.568946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.568948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.569034 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.569368 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.569371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.569372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.569552 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.569637 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.570402 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.570520 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":150,"threads_started":1}
W20260927 00:43:35.570720 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.571575 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.572300 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.571979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37872)
0927 00:43:35.572089 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.572143 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:43:35.573060 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.572833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37878)
0927 00:43:35.572963 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.573009 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:43:35.573165 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.604382 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 1
I20260927 00:43:35.605252 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.604949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37886)
0927 00:43:35.605104 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.605169 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:43:35.605967 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.605713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37890)
0927 00:43:35.605813 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.605860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 00:43:35.606082 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.620296 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 1
I20260927 00:43:35.621222 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.620891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37896)
0927 00:43:35.621069 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.621139 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 00:43:35.621958 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.621737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37902)
0927 00:43:35.621840 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.621894 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:43:35.622078 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.681259 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 2
W20260927 00:43:35.681427 23185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:43:35.681811 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:35.681871 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:35.681924 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:43:35.682152 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.681831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37914)
0927 00:43:35.682000 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.682071 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:43:35.683027 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.682681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37916)
0927 00:43:35.682886 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.682947 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 00:43:35.683192 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
W20260927 00:43:35.686875 23185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:35.686977 23185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:35.687000 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:35.687019 23185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:35.687034 23185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:35.690261 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 2
I20260927 00:43:35.691102 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.690790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37926)
0927 00:43:35.690963 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.691024 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:43:35.692075 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.691688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37932)
0927 00:43:35.691936 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.691995 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260927 00:43:35.692229 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32909: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32909: connect: Connection refused (error 111)
I20260927 00:43:35.694240 23185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41147,127.0.98.61:34403,127.0.98.60:32909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 23:41:00 UTC on ccycloud.quasar-fsjmoa.root.comops.site
build id 4675480
I20260927 00:43:35.695406 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:35.696666 23185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:35.705924 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:35.705933 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:35.705921 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:35.706331 23185 server_base.cc:1034] running on GCE node
I20260927 00:43:35.707000 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:35.708304 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:35.709525 23185 hybrid_clock.cc:648] HybridClock initialized: now 1790469815709483 us; error 58 us; skew 500 ppm
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:35.713037 23185 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:43:35.714238 23185 webserver.cc:460] Webserver started at http://127.0.98.60:38873/ using document root <none> and password file <none>
I20260927 00:43:35.714859 23185 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:35.714933 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:35.715157 23185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:35.717705 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data/instance:
uuid: "9414de9842f645dba6c853f6f998fa83"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "81b8b8820126931e7c6b4a0e0dbc6c3a"
server_key_iv: "e397a19b5c48d70d52c4580d42dca948"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.718370 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal/instance:
uuid: "9414de9842f645dba6c853f6f998fa83"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "81b8b8820126931e7c6b4a0e0dbc6c3a"
server_key_iv: "e397a19b5c48d70d52c4580d42dca948"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.722781 23185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 00:43:35.725656 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:35.727130 23185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 00:43:35.727325 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal
uuid: "9414de9842f645dba6c853f6f998fa83"
format_stamp: "Formatted at 2026-09-27 00:43:35 on dist-test-slave-xltq"
server_key: "81b8b8820126931e7c6b4a0e0dbc6c3a"
server_key_iv: "e397a19b5c48d70d52c4580d42dca948"
server_key_version: "encryptionkey@0"
I20260927 00:43:35.727448 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:35.755396 23185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:35.759014 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:35.759222 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:35.766364 23185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32909
I20260927 00:43:35.766652 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32909 every 8 connection(s)
I20260927 00:43:35.767408 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:43:35.770349 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:35.771682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu as pid 23185
I20260927 00:43:35.771834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSbnIrP/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469528995939-392-0/minicluster-data/master-2/wal/instance
I20260927 00:43:35.772192 392 external_mini_cluster.cc:1428] Setting key ab9292a82b0cb9345641602427964610
I20260927 00:43:35.775426 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:35.782351 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 3
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:35.783371 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } attempt: 3
I20260927 00:43:35.787550 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.777372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.98.62:41147)
0927 00:43:35.777516 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.777520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.777868 (+ 348us) server_negotiation.cc:407] Connection header received
0927 00:43:35.778057 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.778061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.778112 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.778178 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.778954 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.779430 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.780386 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.780593 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.780669 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.783350 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.783354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.783357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.783383 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.785329 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.785833 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.785836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.785838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.785941 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.786301 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.786304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.786306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.786544 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.786614 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.787256 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.787384 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:35.789770 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.777320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41147 (local address 127.0.0.1:45958)
0927 00:43:35.777740 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.777781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.777958 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.778242 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.778248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.778602 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.778755 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.778763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.779551 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.779554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.780251 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.780262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.780744 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.780750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.780866 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.781305 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.781323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.783199 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.785426 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.785432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.785444 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.785725 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.786050 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.786053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.786055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.786198 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.786813 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.786821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.786981 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.788128 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.788666 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":70,"thread_start_us":180,"threads_started":1}
I20260927 00:43:35.789969 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.773951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.0.98.60:32909)
0927 00:43:35.774477 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.774497 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.774542 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:43:35.775437 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.775463 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.775937 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.776373 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.777267 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.778585 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.779540 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.780061 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.780189 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.782884 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.782890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.782902 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.782929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.785122 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.785626 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.785632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.785633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.785778 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.786107 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.786110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.786112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.786692 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.787984 (+ 1292us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.788121 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.788536 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":148,"threads_started":1}
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:35.791754 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.793673 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.782872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37960)
0927 00:43:35.783026 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.783042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.783123 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.783848 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.783851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.784022 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.784130 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.784137 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.785833 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.785836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.786594 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.786601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.787308 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.787311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.787397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.787895 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.787911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.789581 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.791795 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.791798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.791801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.792092 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.792534 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.792537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.792539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.792645 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.793122 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.793125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.793233 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.793318 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.793505 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 00:43:35.794427 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.783265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.0.98.60:32909)
0927 00:43:35.783623 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.783632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.783645 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:35.783678 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.783680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.783721 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.783815 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.784276 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.785709 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.786868 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.787174 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.789742 (+ 2568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.789859 (+ 117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.789863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.789866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.789886 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.791654 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.792308 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.792312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.792314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.792409 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.792764 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.792768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.792769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.793010 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.794056 (+ 1046us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.794125 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.794239 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260927 00:43:35.795635 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.784098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37962)
0927 00:43:35.784392 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.784408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.784487 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.785812 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.785816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.786020 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.786140 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.786148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.787297 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.787300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.788128 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.788141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.788916 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.788920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.789018 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.789484 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.789499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.791719 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.793691 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.793695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.793698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.793967 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.794460 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.794464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.794466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.794566 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.795109 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.795112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.795219 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.795317 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.795477 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 00:43:35.796255 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.784632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.0.98.60:32909)
0927 00:43:35.785564 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.785573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.785585 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:43:35.785619 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.785622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.785664 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.786409 (+ 745us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.786511 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.787168 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.788301 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.788761 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.788902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.791865 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.791869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.791872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.791896 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.793569 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.794244 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.794248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.794250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.794348 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.794790 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.794794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.794796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.795001 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.795092 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.795930 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.796055 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":98,"threads_started":1}
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:43:35.798741 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: Bootstrap starting.
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:35.802049 23103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:35.802764 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.792771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.0.98.61:34403)
0927 00:43:35.792925 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.792930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.793383 (+ 453us) server_negotiation.cc:407] Connection header received
0927 00:43:35.793598 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.793602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.793653 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.793703 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.794306 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.794955 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.796361 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.796665 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.796736 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.799175 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.799179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.799182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.799209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.800862 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.801444 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.801448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.801450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.801541 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.801810 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.801814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.801815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.802113 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.802213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.802494 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.802612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:43:35.802765 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.792423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34403 (local address 127.0.0.1:43648)
0927 00:43:35.793322 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.793335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.793478 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.793858 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.793862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.794046 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.794176 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.794183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.795280 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.795283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.796196 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.796239 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.796885 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.796888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.797099 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.797494 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.797507 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.799015 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.801046 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.801049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.801052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.801302 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.801631 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.801633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.801635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.801713 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.802255 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.802257 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.802339 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.802429 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.802577 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":748}
I20260927 00:43:35.803520 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:35.803995 23103 log.cc:824] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:35.804567 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.793419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.0.98.62:41147)
0927 00:43:35.793537 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.793541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.793552 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:43:35.793585 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.793588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.793629 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.793692 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.794516 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.795220 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.796335 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.796635 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.796737 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.800434 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.800440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.800442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.800475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.802271 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.802784 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.802787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.802789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.802894 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.803205 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.803208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.803210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.803417 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.803602 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.804308 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.804427 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:35.806460 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: Bootstrap starting.
I20260927 00:43:35.806672 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: No bootstrap required, opened a new log
I20260927 00:43:35.809258 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:35.810326 23178 log.cc:824] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:35.810581 23103 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.810869 23103 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f11b3f51134310adde3903206196da, State: Initialized, Role: FOLLOWER
I20260927 00:43:35.811239 23103 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.811789 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:43:35.812045 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.812721 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: No bootstrap required, opened a new log
I20260927 00:43:35.812858 23103 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:35.816881 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.806098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.98.61:34403)
0927 00:43:35.806209 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.806213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.806226 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:35.806261 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.806264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.806306 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.806383 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.807098 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.807829 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.809082 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.809560 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.809673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.812690 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.812694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.812697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.812727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.814500 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.815061 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.815064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.815066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.815152 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.815463 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.815467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.815469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.815690 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.815870 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.816564 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.816689 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 00:43:35 dist-test-slave-xltq krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:35.820458 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.820722 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9aa29f1dd4b490bbb9b5fd050374a0b, State: Initialized, Role: FOLLOWER
I20260927 00:43:35.821059 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.821516 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
W20260927 00:43:35.821671 23274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:43:35.821714 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:43:35.821952 23274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:35.822381 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:35.820004 23035 master_runner.cc:186] Error getting master registration for 127.0.98.61:34403: OK, instance_id { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" instance_seqno: 1790469815525017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:43:35.823890 23035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:35.824477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:35.825609 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.804011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32909 (local address 127.0.0.1:37966)
0927 00:43:35.804294 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:35.804307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:35.804381 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:35.804915 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:35.804918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:35.805082 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:35.805191 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.805198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.813937 (+ 8739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.813941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.814736 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:35.814750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.815347 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.815350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:35.815424 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.815906 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:35.815924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:35.818006 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:35.823988 (+ 5982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.823992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.823995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.824301 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.824604 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:35.824608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:35.824610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:35.824698 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:35.825135 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:35.825138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:35.825220 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 00:43:35.825301 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:43:35.825452 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260927 00:43:35.826190 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.804267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.0.98.60:32909)
0927 00:43:35.804718 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.804722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.804735 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:35.804766 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.804769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.804809 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.812319 (+ 7510us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.812463 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.813578 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.814878 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.815222 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.816560 (+ 1338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.818365 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.818372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.818375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.822094 (+ 3719us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.823849 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.824416 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.824419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.824421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.824507 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.824790 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.824793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.824795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.825039 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.825855 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.825943 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.826062 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260927 00:43:35.829443 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: Bootstrap starting.
W20260927 00:43:35.830829 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:35.830931 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:35.832723 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:35.833702 23254 log.cc:824] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:35.833981 23076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a0f11b3f51134310adde3903206196da"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45968
I20260927 00:43:35.834273 23076 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:43:35.834401 23076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:35.835850 23076 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.837620 23152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0f11b3f51134310adde3903206196da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b"
I20260927 00:43:35.837962 23152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:35.839440 23152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f11b3f51134310adde3903206196da in term 1.
I20260927 00:43:35.840006 23052 leader_election.cc:302] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0f11b3f51134310adde3903206196da, b9aa29f1dd4b490bbb9b5fd050374a0b; no voters:
I20260927 00:43:35.840520 23263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:35.840772 23263 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [term 1 LEADER]: Becoming Leader. State: Replica: a0f11b3f51134310adde3903206196da, State: Running, Role: LEADER
I20260927 00:43:35.841150 23263 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.842414 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: No bootstrap required, opened a new log
I20260927 00:43:35.843120 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f11b3f51134310adde3903206196da. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.843242 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:35.847338 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.847626 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9414de9842f645dba6c853f6f998fa83, State: Initialized, Role: FOLLOWER
I20260927 00:43:35.847981 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } }
I20260927 00:43:35.849592 23292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:35.846805 23110 master_runner.cc:186] Error getting master registration for 127.0.98.60:32909: OK, instance_id { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" instance_seqno: 1790469815764391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:43:35.850155 23110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:35.850915 23076 leader_election.cc:288] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [CANDIDATE]: Term 1 election: Requested vote from peers b9aa29f1dd4b490bbb9b5fd050374a0b (127.0.98.61:34403), 9414de9842f645dba6c853f6f998fa83 (127.0.98.60:32909)
I20260927 00:43:35.851577 23292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:35.855170 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0f11b3f51134310adde3903206196da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9414de9842f645dba6c853f6f998fa83"
I20260927 00:43:35.855485 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:35.856904 23228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f11b3f51134310adde3903206196da in term 1.
I20260927 00:43:35.858623 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.858779 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.858880 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:43:35.873569 23312 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:35.873682 23312 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:35.875301 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.853233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.62:41147)
0927 00:43:35.853343 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.853348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.854637 (+ 1289us) server_negotiation.cc:407] Connection header received
0927 00:43:35.854823 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.854826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.854878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.854929 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.857963 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.858756 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.862031 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.862458 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.862534 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.869114 (+ 6580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.869119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.869122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.869155 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.871100 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.872855 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.872859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.872862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.872964 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.873257 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.873261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.873263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.873518 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.873636 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.874992 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.875135 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:43:35.877229 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.854150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.61:34403)
0927 00:43:35.854258 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.854262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.856001 (+ 1739us) server_negotiation.cc:407] Connection header received
0927 00:43:35.856875 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.856879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.856933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.856983 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.860059 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.860854 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.863251 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.863632 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.863730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.869545 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.869550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.869553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.869585 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.871921 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.873850 (+ 1929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.873854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.873856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.873958 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.874523 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.874526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.874528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.874748 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.874854 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.876929 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.877067 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:43:35.880234 23228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [term 1 FOLLOWER]: Refusing update from remote peer a0f11b3f51134310adde3903206196da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:35.880406 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:35.853800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.0.98.60:32909)
0927 00:43:35.853946 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:35.853950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:35.861390 (+ 7440us) server_negotiation.cc:407] Connection header received
0927 00:43:35.861567 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:35.861570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:35.861619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:35.861665 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:35.862691 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.863359 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.865068 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:35.865409 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:35.865478 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:35.875372 (+ 9894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:35.875377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:35.875380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:35.875411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:35.877161 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.879000 (+ 1839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.879005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.879007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.879112 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:35.879430 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:35.879434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:35.879436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:35.879660 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:35.879782 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:35.880104 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 00:43:35.880261 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:43:35.880935 23290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:35.882205 23152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [term 1 FOLLOWER]: Refusing update from remote peer a0f11b3f51134310adde3903206196da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:35.884373 23290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:35.888686 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f11b3f51134310adde3903206196da. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.888841 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.889962 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f11b3f51134310adde3903206196da. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.890077 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.902714 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.902874 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:35.903010 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.903084 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:35.903185 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.903304 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.904955 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0f11b3f51134310adde3903206196da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0f11b3f51134310adde3903206196da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41147 } } peers { permanent_uuid: "b9aa29f1dd4b490bbb9b5fd050374a0b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34403 } } peers { permanent_uuid: "9414de9842f645dba6c853f6f998fa83" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32909 } } }
I20260927 00:43:35.905088 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:35.918882 23314 mvcc.cc:205] Tried to move back new op lower bound from 7333764365796208640 to 7333764365689188352. Current Snapshot: MvccSnapshot[applied={T|T < 7333764365796208640 or (T in {7333764365796208640})}]
I20260927 00:43:35.919194 23315 mvcc.cc:205] Tried to move back new op lower bound from 7333764365796208640 to 7333764365689188352. Current Snapshot: MvccSnapshot[applied={T|T < 7333764365796208640 or (T in {7333764365796208640})}]
I20260927 00:43:35.923391 23292 catalog_manager.cc:1347] Generated new cluster ID: 4a03b69fc88648288d56956a973545f0
I20260927 00:43:35.923507 23292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:35.940033 23292 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:35.948069 23292 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:35.975492 23292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0f11b3f51134310adde3903206196da: Generated new TSK 0
I20260927 00:43:35.976475 23292 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:43:36.027370 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.024956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.98.62:41147)
0927 00:43:36.025102 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.025107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.026329 (+ 1222us) server_negotiation.cc:407] Connection header received
0927 00:43:36.026495 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.026499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.026558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.026639 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.027211 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45984: BlockingRecv error: recv got EOF from 127.0.0.1:45984 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:43:36.028834 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.024466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.98.61:34403)
0927 00:43:36.024630 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.024634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.027664 (+ 3030us) server_negotiation.cc:407] Connection header received
0927 00:43:36.027719 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.027723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.027776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.027847 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.028620 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43670: BlockingRecv error: recv got EOF from 127.0.0.1:43670 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 00:43:36.030305 23260 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.025817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.0.98.60:32909)
0927 00:43:36.025926 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.025930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.029300 (+ 3370us) server_negotiation.cc:407] Connection header received
0927 00:43:36.029412 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.029415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.029484 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.029557 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.030177 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37988: BlockingRecv error: recv got EOF from 127.0.0.1:37988 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 00:43:36.039687 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.038008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.98.61:34403)
0927 00:43:36.038141 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.038146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.038500 (+ 354us) server_negotiation.cc:407] Connection header received
0927 00:43:36.038758 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.038770 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.038862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.038921 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.039558 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43674: BlockingRecv error: recv got EOF from 127.0.0.1:43674 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:43:36.045627 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.043744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.0.98.61:34403)
0927 00:43:36.043869 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.043874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.044539 (+ 665us) server_negotiation.cc:407] Connection header received
0927 00:43:36.044759 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.044763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.044823 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.044876 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.045481 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43678: BlockingRecv error: recv got EOF from 127.0.0.1:43678 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:43:36.052011 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.050473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.0.98.61:34403)
0927 00:43:36.050586 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.050591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.051063 (+ 472us) server_negotiation.cc:407] Connection header received
0927 00:43:36.051189 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.051192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.051249 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.051338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.051885 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43686: BlockingRecv error: recv got EOF from 127.0.0.1:43686 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 00:43:36.060482 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.058429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.0.98.61:34403)
0927 00:43:36.058656 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.058661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.059080 (+ 419us) server_negotiation.cc:407] Connection header received
0927 00:43:36.059384 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.059388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.059544 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.059669 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.060365 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43692: BlockingRecv error: recv got EOF from 127.0.0.1:43692 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260927 00:43:36.072815 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.071483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.98.61:34403)
0927 00:43:36.071645 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.071649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.071796 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:43:36.071969 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.071973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.072029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.072111 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.072692 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43694: BlockingRecv error: recv got EOF from 127.0.0.1:43694 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 00:43:36.092294 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.091038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.61:34403)
0927 00:43:36.091194 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.091197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.091382 (+ 185us) server_negotiation.cc:407] Connection header received
0927 00:43:36.091545 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.091548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.091602 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.091675 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.092134 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43696: BlockingRecv error: recv got EOF from 127.0.0.1:43696 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:43:36.127526 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.126450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.0.98.61:34403)
0927 00:43:36.126596 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.126600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.126821 (+ 221us) server_negotiation.cc:407] Connection header received
0927 00:43:36.126888 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.126890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.126948 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.127038 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.127419 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43712: BlockingRecv error: recv got EOF from 127.0.0.1:43712 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 00:43:36.194790 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.193666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.98.61:34403)
0927 00:43:36.193802 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.193806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.193924 (+ 118us) server_negotiation.cc:407] Connection header received
0927 00:43:36.194052 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.194054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.194106 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.194189 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.194679 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43724: BlockingRecv error: recv got EOF from 127.0.0.1:43724 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:43:36.326224 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.324958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.98.61:34403)
0927 00:43:36.325159 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.325164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.325403 (+ 239us) server_negotiation.cc:407] Connection header received
0927 00:43:36.325528 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.325531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.325588 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.325680 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.326111 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43728: BlockingRecv error: recv got EOF from 127.0.0.1:43728 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 00:43:36.585675 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:43:36.584478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.98.61:34403)
0927 00:43:36.584618 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:36.584623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:36.584783 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:43:36.584933 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:36.584937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:36.584997 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:36.585079 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:36.585564 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43730: BlockingRecv error: recv got EOF from 127.0.0.1:43730 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:43:36.832494 23287 catalog_manager.cc:1259] Loaded cluster ID: 4a03b69fc88648288d56956a973545f0
I20260927 00:43:36.832590 23287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: loading cluster ID for follower catalog manager: success
I20260927 00:43:36.834576 23287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: acquiring CA information for follower catalog manager: success
I20260927 00:43:36.835920 23287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9aa29f1dd4b490bbb9b5fd050374a0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:36.875049 23312 catalog_manager.cc:1259] Loaded cluster ID: 4a03b69fc88648288d56956a973545f0
I20260927 00:43:36.875146 23312 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: loading cluster ID for follower catalog manager: success
I20260927 00:43:36.876957 23312 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: acquiring CA information for follower catalog manager: success
I20260927 00:43:36.878196 23312 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9414de9842f645dba6c853f6f998fa83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:37.104864 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:37.100134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.98.61:34403)
0927 00:43:37.100446 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:37.100450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:37.100468 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:43:37.100627 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:37.100629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:37.100759 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:37.100876 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:37.101261 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.101800 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.102813 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.103065 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.103169 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:37.103669 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:37.104308 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:37.104438 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:37.104631 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 00:43:37.104699 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":121,"threads_started":1}
I20260927 00:43:37.113089 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:37.108514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.0.98.60:32909)
0927 00:43:37.108759 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:37.108763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:37.108957 (+ 194us) server_negotiation.cc:407] Connection header received
0927 00:43:37.109170 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:37.109173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:37.109313 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:37.109444 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:37.109722 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.110321 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.111347 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.111597 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.111702 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:37.112122 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:37.112652 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:37.112760 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:37.112880 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 00:43:37.112945 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260927 00:43:37.122390 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:37.117298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.98.61:34403)
0927 00:43:37.117556 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:37.117561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:37.118250 (+ 689us) server_negotiation.cc:407] Connection header received
0927 00:43:37.118487 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:37.118491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:37.118564 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:37.118654 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:37.119028 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.119686 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121019 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.121401 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121512 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:37.121807 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:37.121942 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:37.122009 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:37.122109 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 00:43:37.122191 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260927 00:43:37.122627 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:37.118011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.0.98.60:32909)
0927 00:43:37.118211 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:37.118216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:37.118292 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:43:37.118827 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:37.118830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:37.118892 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:37.118974 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:37.119260 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.120101 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121318 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.121653 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121752 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:37.122076 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:37.122190 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:37.122233 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:37.122390 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:43:37.122465 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:43:37.123255 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:37.117299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.98.62:41147)
0927 00:43:37.117557 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:37.117561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:37.117918 (+ 357us) server_negotiation.cc:407] Connection header received
0927 00:43:37.118131 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:37.118135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:37.118402 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:37.118571 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:37.118952 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.119910 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121104 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:37.121444 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:37.121552 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:37.122078 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:37.122478 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:37.122545 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:37.123030 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 00:43:37.123107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260927 00:43:37.127982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 23035
I20260927 00:43:37.134497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 23110
I20260927 00:43:37.141770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSbnIrP/build/debug/bin/kudu with pid 23185
2026-09-27T00:43:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3100 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6167 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288137 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST